staging: Deploy maven indexer/lister/loader

When the actual diffs land, making it run in staging should help in lifting papercuts.

Plan:

• docker declaration should be complete now, make it run within docker

• #3746 (closed), D7023: unstuck the listing scheduling

• D7052: Land

• Package new version

• #3746 (closed): Failing with a new issue

• D7139: Fix that issue

• package new version v2.6.3

• Make the lister actually list "maven" (load-maven) tasks to load

• D7178: Make the scheduler actually schedule load-maven tasks

• #4105 (closed): "Industrialize" maven-index-exporter docker image

• Develop puppet manifest for the maven stack (indexer, lister, loader)

  • infra/puppet/puppet-swh-site!505: Deploy node with maven index exporter service which computes the expected lister output (export.fld)
  • infra/puppet/puppet-swh-site!506: Expose the export files through apache
  • infra/puppet/puppet-swh-site!507: Update lister service to also manage the list maven
  • infra/puppet/puppet-swh-site!508: Deploy swh-worker@loader_maven service (in charge of dealing with jar files)
  • Install zfs tooling on that node (to reduce future disk space use [2])

• infra/puppet/puppet-swh-site!507: Provision the new node to expose the computation results

• Ensure the loader and lister maven are registered on the staging scheduler [1]

• swh-sysadmin-provisioning!69 (closed): Provision new node

• Update inventory

• Configure zfs partitions

• Update firewall rules to allow icinga reporting

• Finally schedule new maven lister instance to consume maven-central and clojars (for now)

• [1]

$ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven
INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven

• [2] export.fld can be quite large (maven-central ~18G), others might be less though clojars ~60Mib)

---

Migrated from T3746 (view on Phabricator)