Deploy opam lister/loader to staging

Plan:

• lister v0.15 (includes opam lister)

• loader.core: Tag v0.23 which includes the opam loader

• Register opam listing and loading task in the scheduler [1]

• infra/puppet/puppet-swh-site!386: Add opam lister task to the worker configuration

• Upgrade loader.core to staging workers and restart swh-worker@lister service

• Trigger first listing [2]

• Monitoring

• infra/puppet/puppet-swh-site!387: Puppetize new swh-worker@loader_opam

• Deploy new loader

• Start loader service

• [1]

swhscheduler@scheduler0:~$ dpkg -l python3-swh.loader.core
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                    Version               Architecture Description
+++-=======================-=====================-============-=================================
ii  python3-swh.loader.core 0.23.1-1~swh1~bpo10+1 all          Software Heritage Loader Core
swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
...
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam
INFO:swh.scheduler.cli.task_type:Create task type list-opam in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam
INFO:swh.scheduler.cli.task_type:Create task type load-opam in scheduler
...
$ psql service=staging-swh-scheduler
16:49:33 swh-scheduler@db1:5432=> select * from task_type where type like '%opam%';
+-----------+-----------------------------------+----------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
|   type    |            description            |              backend_name              | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
+-----------+-----------------------------------+----------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
| list-opam | Lister task for the Opam registry | swh.lister.opam.tasks.OpamListerTask   | 1 day            | 1 day        | 1 day        |              1 |           (null) |      (null) | (null)      |
| load-opam | Load Opam's artifacts             | swh.loader.package.opam.tasks.LoadOpam | 1 day            | 1 day        | 1 day        |              1 |           (null) |      (null) | (null)      |
+-----------+-----------------------------------+----------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
(2 rows)

Time: 144.033 ms

• [2]

swhscheduler@scheduler0:~$ /usr/bin/swh scheduler --config-file scheduler.yml task add list-opam url=https://opam.ocaml.org instance=opam.ocaml.org
Created 1 tasks

Task 24858921
  Next run: today (2021-08-05T15:20:39.703665+00:00)
  Interval: 1 day, 0:00:00
  Type: list-opam
  Policy: recurring
  Args:
  Keyword args:
    instance: 'opam.ocaml.org'
    url: 'https://opam.ocaml.org'

---

Migrated from T3438 (view on Phabricator)