production: Migrate rpc counter to dynamic infrastructure
List of services to migrate:
- gunicorn-swh-counters
- swh-counters-journal-client
gunicorn-swh-counters access to:
- counters1: redis backend. Not to be migrated. Stays on that node.
- counter1: /srv/softwareheritage/counters holding *.json historic dataset
journal client access to:
- counter1: redis backend
Plan:
- swh/infra/ci-cd/swh-charts!334 (merged): swh-charts: Deploy journal client to dynamic infra
- Deploy
- Deactivate journal client in counter1
- (manual) counters1: Decommission journal client
-
Checks
- https://grafana.softwareheritage.org/goto/3V4L2M2Sz?orgId=1: grafana board
- There shall be no lag
- Historic data served by gitlab in swh.counters repository as snippet
- swh/infra/ci-cd/swh-charts!335 (merged): cronjob: Add counter cronjob to refresh periodically cache (conditional).
- swh/infra/ci-cd/swh-charts!336 (closed): swh-charts: Migrate rpc service to dynamic infra
-
Adapt redis backend so it can be remotely accessed by the pods in the cluster (if needed)(already done with staging) - Deploy counters rpc service to dynamic infra
- Adapt firewall rule so workers can access thanos.internal.admin.swh.network
- swh/infra/ci-cd/swh-charts!337 (merged): swh-charts: Adapt dependent services to use the new counters rpc
- (manual) Stop gunicorn-swh-counters
- Checks
- swh/devel/swh-docs!407 (merged): docs: Update service-urls reference [2]
Post-migration:
- swh/infra/puppet/puppet-swh-site!691 (merged): Decommission counter1 swh services, keep only the redis running
[2] https://docs.softwareheritage.org/sysadm/network-architecture/service-urls.html#public-urls-1
Refs. #5179 (closed)
Refs. #5110 (closed)