production: Migrate search workload to dynamic infra

A current read-only search instance is deployed in the elastic infra.

Remains to:

• migrate search journal clients (swh-search-journal-client@indexed.service, swh-search-journal-client@objects.service)
• Adapt search instance to receive writes (reuse existing one, probably need to adapt its workers/threads/memory)
• (manual) Stop journal client on search1.internal.softwareheritage.org and check for lag [1] ~> no lag
• Decommission search1.internal.softwareheritage.org
• swh/devel/swh-docs!405 (merged): Update service-urls documentation

[1] https://grafana.softwareheritage.org/goto/-XdtuX5Ik?orgId=1

Legend:

• already done

Refs. #5110 (closed)

marked this issue as related to #5110 (closed)

mentioned in issue #5110 (closed)

marked the checklist item migrate search journal clients as completed

marked the checklist item swh-search-journal-client@indexed.service as completed

marked the checklist item swh-search-journal-client@objects.service as completed

marked the checklist item Adapt search instance to receive writes (reuse existing one, probably need to adapt its workers/threads/memory) as completed

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@cfd78a13

mentioned in merge request swh/infra/puppet/puppet-swh-site!686 (merged)

changed the description

marked the checklist item Decommission search1.internal.softwareheritage.org as completed

closed

mentioned in commit swh/infra/ci-cd/swh-charts@9e16cd37