Migrate production "static" worker to "elastic" workers

wip (feel free to amend) [3]

Blockers:

• logs pushed to elasticsearch
• Recycle 3 ip public addresss from static workers to give to rancher-metal0[1-3] (not a blocker so far)

Ideal but not blocking:

• Alerting (if something is not running properly [1])
• #4819 (closed): Priority on services (for kube scheduler in case of pressure) - charts ready but not yet activated on production pods.

Regarding the charts, everything already exists since it's eployed in staging already.

Plan:

• Migrate listers
  • possible special case for github lister [2]
• Migrate loaders
  • high concurrency for git
• Migrate indexers (running on azure today) (we'll keep them running there for now)
• Migrate deposit checker
• Decommission worker nodes
• swh/infra/puppet/puppet-swh-site!644 (merged): swh-site: Update manifest to decommission the nodes too.

[1] Something already exists but it's not widely deployed

[2] Nothing might be required though. We can declare that lister as a standalone one (to double check). Currently, it's running in a dedicated vm to avoid starvation on the other listers.

[3] https://hedgedoc.softwareheritage.org/4ZHT03kRT7mHYOEm1MSueQ#Points-to-solve-before-going-to-production

changed the description

changed the description

marked the checklist item Migrate listers as completed

marked the checklist item possible special case for github lister [2] as completed

marked the checklist item Migrate loaders as completed

marked the checklist item high concurrency for git as completed

marked the checklist item Alerting (if something is not running properly [1]) as completed

The decommissioning of the static nodes and termination of remaining tasks is still in progress, but everything is (also) deployed in the production k8s cluster.

We aligned all listers and loaders which were running in the static infra on the elastic infra. No issues.

This even lead to improve the configmap check to only trigger redeployment of actual impacted lister or loader (it was not the case before).

marked the checklist item Migrate deposit checker as completed

mentioned in commit swh/infra/ci-cd/swh-charts@5ea8fb5d

mentioned in commit swh/infra/ci-cd/swh-charts@ac39ec5f

mentioned in commit swh/infra/ci-cd/swh-charts@e92c50da

marked the checklist item Priority on services (for kube scheduler in case of pressure) - charts ready but not yet activated on production pods. as completed

changed the description

marked this issue as related to #5053 (closed)

mentioned in commit swh/infra/puppet/puppet-swh-site@0664b463

mentioned in commit swh/infra/puppet/puppet-swh-site@6bdca102

mentioned in merge request swh/infra/puppet/puppet-swh-site!644 (merged)

mentioned in commit swh/infra/puppet/puppet-swh-site@c1849c49

changed the description

marked the checklist item swh/infra/puppet/puppet-swh-site!644 (merged): swh-site: Update manifest to decommission the nodes too. as completed

closed