Skip to content

staging: Deploy pubdev

Implies:

  • new docker image release including swh.lister v3.0
  • new docker image release including swh.loader.core v4.0
  • Register new task types to the staging scheduler [1]
  • Missing load-pubdev task type in scheduler -> reason: python3-swh.loader.core v4.0 debian build broken
  • Fix debian unstable build ^ as it's needed for the scheduler node to register all new task types [1]
  • Fix yet another debian build (buster) [2] -> False negative (dns issue during build)
  • Register load-pub task type (among other loader tasks) [1] (amended)
  • Schedule one task with type 'list-pubdev' [3]
  • infra/swh-charts!11: archive-staging cluster: declare new loader and lister
  • Checks:
    • listing ok [4]
    • loading ok

Note:

  • Use new elastic worker infra

  • [1]

swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
...
INFO:swh.scheduler.cli.task_type:Create task type load-crates in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-crates in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler
...
# ^^^^ missing couple of loading tasks -> probably an issue with the debian package build

swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core | grep ii
ii  python3-swh.loader.core 4.0.0-1~swh2~bpo10+1 all          Software Heritage Loader Core
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register  | grep pubdev
...
INFO:swh.scheduler.cli.task_type:Create task type load-arch in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-aur in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-pubdev in scheduler
...
swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-pubdev
Created 1 tasks

Task 33419442
  Next run: today (2022-09-09T10:30:00.252081+00:00)
  Interval: 1 day, 0:00:00
  Type: list-pubdev
  Policy: recurring
  Args:
  Keyword args:
  • [4]
12:37:31 swh-scheduler@db1:5432=> select id, name, instance_name from listers where name='pubdev';
+--------------------------------------+--------+---------------+
|                  id                  |  name  | instance_name |
+--------------------------------------+--------+---------------+
| a299a643-0887-44d1-8d71-88da0e1e3b52 | pubdev | pubdev        |
+--------------------------------------+--------+---------------+
(1 row)

Time: 6.813 ms
12:37:29 swh-scheduler@db1:5432=> select now(), visit_type, count(*) from listed_origins where lister_id='a299a643-0887-44d1-8d71-88da0e1e3b52' group by visit_type;
+-------------------------------+------------+-------+
|              now              | visit_type | count |
+-------------------------------+------------+-------+
| 2022-09-09 10:37:31.624512+00 | pubdev     |  1000 |
+-------------------------------+------------+-------+
(1 row)

Time: 7.084 ms
  • [5]
│ loaders [2022-09-09 10:42:40,267: INFO/MainProcess] Connected to amqp://swhconsumer:**@scheduler0.internal.staging.swh.network:5672//                                                                                                       │
│ loaders [2022-09-09 10:42:40,823: INFO/MainProcess] loader@loader-pubdev-d699448b7-46dc5 ready.                                                                                                                                             │
│ loaders [2022-09-09 10:44:29,827: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] received                                                                                          │
│ loaders [2022-09-09 10:44:29,872: INFO/MainProcess] Task swh.loader.package.pubdev.tasks.LoadPubDev[388f30f6-6b6f-423f-aa25-f3f5a809d245] received                                                                                          │
│ loaders [2022-09-09 10:44:37,388: INFO/ForkPoolWorker-1] Task swh.loader.package.pubdev.tasks.LoadPubDev[0a0fba7e-97f1-4b6e-b6bc-e4e8190f0897] succeeded in 7.445372300921008s: {'status': 'eventful', 'snapshot_id': '0dab696ae4f51cf95e44 │

Migrated from T4517 (view on Phabricator)

Edited by Antoine R. Dumont