staging: Deploy deposit rpc service

• Adapt configuration to allow use of specific backend (e.g. with django-storges extension, either azure [1] or s3 [2] ...)
• swh-apps: Adapt images with the proper dependency
• swh/infra/ci-cd/swh-charts!197 (merged): Adapt chart to override default swh.deposit configuration which uses a specific configuration
• swh/devel/snippets@a34da1cd: Write migration script to upload existing deposit client tarballs to the new backend
• staging: Label nodes with swh/deposit [1]
• Migrate existing data from static staging deposit to dynamic staging deposit
• Activate such staging deposit chart in staging: fail the deposit assumes that deposit tarballs are local to itself (needs adaptation)
• (review) swh/devel/swh-deposit!419 (merged): Adapt deposit so it can check remote tarballs
• swh/devel/swh-deposit!419 (comment 156200): Run deposit within the docker-dev with equivalent remote backend (azurite)
• (review) swh/devel/swh-deposit!419 (merged): Adapt deposit configuration so we can use remote backend too
• (pending review) swh/infra/ci-cd/swh-charts!225 (merged): Simplify chart configuration
• swh/infra/ci-cd/swh-charts!224 (merged): Deploy next version
• Checks: Issue with deposit server (when reading tarball files)
• Fix deposit server issue (static)
• swh/infra/ci-cd/swh-charts!227 (merged): elastic infra: Deploy next version
• Checks (icinga) -> failing...
• Troubleshoot issue: wrong headers stored in the blobstorage which prevents the archive reading (it fails with ContentDecodingError)
• Fix
  • swh/devel/swh-deposit!421 (merged): Allow extra azure configuration in settings.production
  • Merge and tag
  • swh/infra/ci-cd/swh-charts!240 (merged): Deploy new version
  • swh/infra/ci-cd/swh-charts!238 (merged): Configure extra azure configuration for the elastic deposit
• Status
• swh/infra/puppet/puppet-swh-site!662 (merged): Migrate reverse proxy to hit the elastic deposit (through puppet, not manually)
• swh/infra/puppet/puppet-swh-site!663 (merged): Migrate puppet collectable icinga checks to static checks
• Decommission deposit.staging services
• Checks
  • Fix icinga checks (throttling) [4]
  • icinga (pergamon): Deploy new icinga plugin version
  • icinga check ok

optional:

• Adapt docker-dev configuration with deposit with remote backends

bonus:

• swh/devel/swh-deposit!419 (merged): Adapt swh.deposit directly

[1] https://django-storages.readthedocs.io/en/latest/backends/azure.html

[2] https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html

[3]

$ environment=staging; for i in $(seq 1 6); do for label in deposit; do kubectl --context archive-$environment-rke2 label --overwrite node rancher-node-staging-rke2-worker$i swh/$label=true; done; done
node/rancher-node-staging-rke2-worker1 labeled
node/rancher-node-staging-rke2-worker2 labeled
node/rancher-node-staging-rke2-worker3 labeled
node/rancher-node-staging-rke2-worker4 labeled
node/rancher-node-staging-rke2-worker5 labeled
node/rancher-node-staging-rke2-worker6 labeled
$ environment=production; for i in $(seq 1 3); do for label in deposit; do kubectl --context archive-$environment-rke2 label --overwrite node rancher-node-metal0$i swh/$label=true; done; done
node/rancher-node-metal01 not labeled
node/rancher-node-metal02 not labeled
node/rancher-node-metal03 not labeled

Note: @gsamson ^

[4] swh/devel/swh-icinga-plugins!55 (merged)

Refs. #4780 (closed)