staging: Deploy latest nixguix lister and new vcs directory loaders

Contains:

• New hg, git, svn directory loader able to ingest those vcs trees
• New swh.loader.{core,git,svn,hg} version
• New swh.lister version

Plan:

• Tag modules and build packages
  • v5.6.1: lister
  • v5.6.0: loader.core
  • v2.4.0: loader.git
  • v3.3.0: loader.hg
  • v1.8.0: loader.svn [1]
  • v5.6.2: lister
  • v5.6.3: lister
  • v5.7.1: loader.core
• Build images
  • swh-lister
  • swh-loader-savecodenow
  • swh-loader-git
  • swh-loader-mercurial
  • swh-loader-svn
  • swh-lister
  • swh-lister
  • swh-loader-savecodenow

staging:

• swh-charts: Add new Directory loader deployment (consuming the new queues [1])
• swh-charts: Update image versions to deploy
• scheduler0.staging 'scheduler' db: Clean up listed origins with visit_type 'directory'
• Deploy
• scheduler0.staging: Upgrade impacted python3 packages
• scheduler0.staging: Register new task types [2]
• scheduler0.staging: Restart swh-scheduler-schedule-recurrent service (to pick up the new task type)
• Checks
  • listing ok
  • ingestion ok? so far, NO!
    • 1. no, wrong docker image used in the chart (fixed)
    • 2. no, visit type and task name discrepancy makes for no scheduling (fixed through sql query in db)
    • 3: no, mercurial queue name is not ok (fixed)
    • 4. no, the lister does not propagate the reference it read from the upstream manifest (fixed)
    • 5. no, the tarball directory loader's visit type is inconsistent with the task name, scheduling issue again (fixed)
  • scheduled origins are ingested (some work, some don't, sentry does catch those)
    • svn-export
    • hg-checkout
    • git-checkout
    • tarball-directory
  • ingestion on those origins is ok

[1] swh.loader.{git.tasks.LoadGit,svn.tasks.LoadSvn,hg.tasks.LoadHg}Directory

[2] load-{git,svn,hg}-directory

[3] debian build is unhappy though, not that it's too much blocking for staging

Refs. swh/meta#4979 (closed)

changed the description

mentioned in issue swh/meta#4979 (closed)

changed the description

changed the description

changed the description

changed the description

changed the description

changed the description

marked the checklist item swh-lister as completed

marked the checklist item swh-loader-git as completed

marked the checklist item swh-loader-mercurial as completed

marked the checklist item swh-loader-svn as completed

changed the description

[ ] scheduler0.staging 'scheduler' db: Clean up listed origins with visit_type 'directory'

The plan would be the following.

Data to cleanup [1].

Actual cleanup [2] on the staging scheduling db should do. Simpler to delete than to actually pinpoint the 'directory' visit types to drop (some will remain legit, the tarball), the others won't be.

[2]

-- disable the listing tasks
update task set status='disabled' where type='list-nixguix';
-- clean up old origins
delete from listed_origins where lister_id=(select id from listers where name='nixguix' );
-- trigger back a listing
update task set status='next_run_not_scheduled', next_run=now() where type='list-nixguix';

[1]

2023-06-06 16:13:34 swh-scheduler@db1:5432 λ select visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='nixguix' )  group by visit_type;
+------------+-------+
| visit_type | count |
+------------+-------+
| content    |  4397 |
| directory  | 18915 |
| git        |  6906 |
| hg         |    27 |
| svn        |   117 |
+------------+-------+
(5 rows)

Time: 961.939 ms

marked the checklist item swh-loader-savecodenow as completed

marked the checklist item Build images as completed

mentioned in commit swh/infra/ci-cd/swh-charts@b7798551

mentioned in commit swh/infra/ci-cd/swh-charts@a178155b