staging: Deploy maven indexer/lister/loader
When the actual diffs land, making it run in staging should help in lifting papercuts.
Plan:
-
docker declaration should be complete now, make it run within docker -
#3746 (closed), D7023: unstuck the listing scheduling -
D7052: Land -
Package new version -
#3746 (closed): Failing with a new issue -
D7139: Fix that issue -
package new version v2.6.3 -
Make the lister actually list "maven" (load-maven) tasks to load -
D7178: Make the scheduler actually schedule load-maven tasks -
#4105 (closed): "Industrialize" maven-index-exporter docker image -
Develop puppet manifest for the maven stack (indexer, lister, loader) -
infra/puppet/puppet-swh-site!505: Deploy node with maven index exporter service which computes the expected lister output (export.fld) -
infra/puppet/puppet-swh-site!506: Expose the export files through apache -
infra/puppet/puppet-swh-site!507: Update lister service to also manage the list maven -
infra/puppet/puppet-swh-site!508: Deploy swh-worker@loader_maven service (in charge of dealing with jar files) -
Install zfs tooling on that node (to reduce future disk space use [2])
-
-
infra/puppet/puppet-swh-site!507: Provision the new node to expose the computation results -
Ensure the loader and lister maven are registered on the staging scheduler [1] -
infra/swh-sysadmin-provisioning!69: Provision new node -
Update inventory -
Configure zfs partitions -
Update firewall rules to allow icinga reporting -
Finally schedule new maven lister instance to consume maven-central and clojars (for now) -
[1]
$ swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler/backend.yml task-type register | grep -i maven
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin loader.maven
INFO:swh.scheduler.cli.task_type:Create task type load-maven in scheduler
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.maven- [2] export.fld can be quite large (maven-central ~18G), others might be less though clojars ~60Mib)
Migrated from T3746 (view on Phabricator)