Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • S sysadm-environment
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 167
    • Issues 167
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Platform
  • Infrastructure
  • sysadm-environment
  • Issues
  • #4517
Closed
Open
Issue created Sep 09, 2022 by Antoine R. Dumont@ardumontOwner12 of 12 checklist items completed12/12 checklist items

staging: Deploy pubdev

Implies:

  • new docker image release including swh.lister v3.0
  • new docker image release including swh.loader.core v4.0
  • Register new task types to the staging scheduler [1]
  • Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken
  • Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1]
  • Fix yet another debian build (buster) [2] -> False negative (dns issue during build)
  • Register load-pub task type (among other loader tasks) [1] (amended)
  • Schedule one task with type 'list-pubdev' [3]
  • swh-charts!11: archive-staging cluster: declare new loader and lister
  • Checks:
    • listing ok [4]
    • loading ok

Note:

  • Use new elastic worker infra

  • [1]

swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
...
INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler
...
# ^^^^ missing couple of loading tasks -> probably an issue with the debian package build

swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core | grep ii
ii  python3-swh.loader.core 4.0.0-1~swh2~bpo10+1 all          Software Heritage Loader Core
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register  | grep pubdev
...
INFO:swh.scheduler.cli.task_type:Create task type load-arch in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-aur in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-pubdev in scheduler
...
  • [1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/286/console

  • [2] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DLDBASE/job/gbp-buildpackage/288/console

  • [3]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-pubdev
Created 1 tasks

Task 33419442
  Next run: today (2022-09-09T10:30:00.252081+00:00)
  Interval: 1 day, 0:00:00
  Type: list-pubdev
  Policy: recurring
  Args:
  Keyword args:
  • [4]
12:37:31 swh-scheduler@db1:5432=> select id, name, instance_name from listers where name='pubdev';
+--------------------------------------+--------+---------------+
|                  id                  |  name  | instance_name |
+--------------------------------------+--------+---------------+
| a299a643-0887-44d1-8d71-88da0e1e3b52 | pubdev | pubdev        |
+--------------------------------------+--------+---------------+
(1 row)

Time: 6.813 ms
12:37:29 swh-scheduler@db1:5432=> select now(), visit_type, count(*) from listed_origins where lister_id='a299a643-0887-44d1-8d71-88da0e1e3b52' group by visit_type;
+-------------------------------+------------+-------+
|              now              | visit_type | count |
+-------------------------------+------------+-------+
| 2022-09-09 10:37:31.624512+00 | pubdev     |  1000 |
+-------------------------------+------------+-------+
(1 row)

Time: 7.084 ms
  • [5]
│ loaders [2022-09-09 10:42:40,267: INFO/MainProcess] Connected to amqp://swhconsumer:**@scheduler0.internal.staging.swh.network:5672//                                                                                                       │
│ loaders [2022-09-09 10:42:40,823: INFO/MainProcess] loader@loader-pubdev-d699448b7-46dc5 ready.                                                                                                                                             │
│ loaders [2022-09-09 10:44:29,827: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] received                                                                                          │
│ loaders [2022-09-09 10:44:29,872: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[388f30f6-6b6f-423f-aa25-f3f5a809d245] received                                                                                          │
│ loaders [2022-09-09 10:44:37,388: INFO/ForkPoolWorker-1] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] succeeded in 7.445372300921008s: {'status': 'eventful', 'snapshot_id': '0dab696ae4f51cf95e44 │

Migrated from T4517 (view on Phabricator)

Edited Oct 18, 2022 by Antoine R. Dumont
Assignee
Assign to
Time tracking