Deploy CVS loader in production

After discussing with @teams/sysadmin, this will be deployed in the elastic infra.

In between the call for review and now, there has been at least one commit to improve the behavior, so might as well bump that version too (-> v0.6).

staging:

• Tag v0.6
• Build new docker image (once the main build is done ^)
  • swh.loader.cvs
  • swh.loader.highpriority
• Deploy new versions
• Checks

production:

• Deploy cvs loader in production
• Adapt cvs highpriority consumption in production
• Checks

scheduling:

• scheduler (saatchi): Register new task type [1] [2] [3]
• Schedule some origins for ingestion [4]
• Ensure recurrent origins are getting scheduled (systemctl status swh-scheduler-schedule-recurrent) [7]

[7] Service got restarted to pick up new cvs type.

root@saatchi:~# systemctl status swh-scheduler-schedule-recurrent.service | grep Active
     Active: active (running) since Mon 2023-03-13 15:06:08 UTC; 12min ago

archive ("savecodenow" ui):

• scheduler: restart scheduler runner priority [5]
• swh-site: Ensure scheduler runner priority is aware of cvs origins (adapt configuration) [6]
• scheduler: restart scheduler runner priority to take care of cvs origins
• webapp1: Ensure the cvs task type is showing up in the archive webapp (after systemctl reload-or-restart gunicorn-swh-webapp)
• Are cvs origins picked up for scheduling yet? Yes
• moma: systemctl reload-or-restart gunicorn-swh-webapp
• Is 'cvs' type a possible selection in the save code now ui? Yes

[1] cvs

swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs
INFO:swh.scheduler.cli.task_type:Create task type load-cvs in scheduler

[2]

2023-03-13 15:12:03 softwareheritage-scheduler@belvedere:5432 λ select * from task_type where type='load-cvs';
+-[ RECORD 1 ]-----+----------------------------------------+
| type             | load-cvs                               |
| description      | Import a CVS repository                |
| backend_name     | swh.loader.cvs.tasks.LoadCvsRepository |
| default_interval | 1 day                                  |
| min_interval     | 1 day                                  |
| max_interval     | 1 day                                  |
| backoff_factor   | 1                                      |
| max_queue_length | 1000                                   |
| num_retries      | (null)                                 |
| retry_delay      | (null)                                 |
+------------------+----------------------------------------+

Time: 4.743 ms

[4]

swhscheduler@saatchi:~$ swh scheduler -C /etc/softwareheritage//scheduler/listener-runner.yml origin send-to-celery --lister-name sourceforge --lister-instance-name main --queue swh.loader.cvs.tasks.LoadCvsRepository cvs
10000 slots available in celery queue
10000 visits to send to celery

[5]

root@saatchi:~# systemctl restart swh-scheduler-runner-priority.service

[6]

$ puppet agent -t ...
-ExecStart=/usr/bin/swh --log-level INFO scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml start-runner --period 10 --with-priority --task-type load-bzr --task-type load-git --task-type load-svn --task-type load-archive-files --task-type load-hg
+ExecStart=/usr/bin/swh --log-level INFO scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml start-runner --period 10 --with-priority --task-type load-bzr --task-type load-cvs --task-type load-git --task-type load-svn --task-type load-archive-files --task-type load-hg

[3] Full details

task type register full output (for later)

swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cvs
INFO:swh.scheduler.cli.task_type:Create task type load-cvs in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.content
INFO:swh.scheduler.cli.task_type:Create task type load-content in scheduler
INFO:swh.scheduler.cli.task_type:Create task type load-directory in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.directory
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.arch
INFO:swh.scheduler.cli.task_type:Create task type load-arch in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.archive
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.aur
INFO:swh.scheduler.cli.task_type:Create task type load-aur in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.conda
INFO:swh.scheduler.cli.task_type:Create task type load-conda in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cpan
INFO:swh.scheduler.cli.task_type:Create task type load-cpan in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.cran
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.crates
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.debian
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.deposit
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.golang
INFO:swh.scheduler.cli.task_type:Create task type load-golang in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.nixguix
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.npm
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.opam
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pubdev
INFO:swh.scheduler.cli.task_type:Create task type load-pubdev in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.puppet
INFO:swh.scheduler.cli.task_type:Create task type load-puppet in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.pypi
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.bzr
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.arch
INFO:swh.scheduler.cli.task_type:Create task type list-arch in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.aur
INFO:swh.scheduler.cli.task_type:Create task type list-aur in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bitbucket
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.bower
INFO:swh.scheduler.cli.task_type:Create task type list-bower in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cgit
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.conda
INFO:swh.scheduler.cli.task_type:Create task type list-conda in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cpan
INFO:swh.scheduler.cli.task_type:Create task type list-cpan in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.cran
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.crates
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.debian
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.github
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitlab
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gnu
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.golang
INFO:swh.scheduler.cli.task_type:Create task type list-golang in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-golang-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.hackage
INFO:swh.scheduler.cli.task_type:Create task type list-hackage in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.nixguix
INFO:swh.scheduler.cli.task_type:Create task type list-nixguix in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.npm
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.nuget
INFO:swh.scheduler.cli.task_type:Create task type list-nuget in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.opam
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.packagist
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.phabricator
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pubdev
INFO:swh.scheduler.cli.task_type:Create task type list-pubdev in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.puppet
INFO:swh.scheduler.cli.task_type:Create task type list-puppet in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.pypi
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.rubygems
INFO:swh.scheduler.cli.task_type:Create task type list-rubygems in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.sourceforge
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.tuleap
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gogs
INFO:swh.scheduler.cli.task_type:Create task type list-gogs-full in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.fedora
INFO:swh.scheduler.cli.task_type:Create task type list-fedora-full in scheduler

added activity::Deployment roadmap_import labels

@ardumont validated for deployment in production :)

moved from swh/meta#4751 (moved)

changed the description

marked the checklist item Build new docker image (once the main build is done ^) as completed

mentioned in commit ardumont/swh-charts@545fa2d0

mentioned in commit ardumont/swh-charts@6dc8d0c6

marked the checklist item Build new docker image (once the main build is done ^) as incomplete

mentioned in commit ardumont/swh-charts@8f6d7fa3

marked the checklist item swh.loader.highpriority as completed

marked the checklist item Build new docker image (once the main build is done ^) as completed

changed the description

marked the checklist item Checks as completed

marked the checklist item Deploy cvs loader in production as completed

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@9a565828

marked the checklist item Adapt cvs highpriority consumption in production as completed