[CEA] Prepare a productized replaying stack

Some constraints:

• Aws public bucket is not reachable from the CEA. We should ask for the whitelisting later when we will have gathered all the other sites we need [1].
• We need some monitoring, nothing is deployed yet on the CEA infra (but it will)
• A redis for the error reporting must be deployed
• The containerized replayers will probably be needed later for the mirrors

The best candidate right now is adapt the swh chart to be able to deployed the replayers in the elastic infra

[1] The li

• https://softwareheritage.s3.amazonaws.com
• public traffic to the kafka cluster broker[1-4?].journal.softwareheritage.org:9093 (need an network configuration update )
• https://objstorage.softwareheritage.org

changed milestone to %Scale-out objstorage in production as primary objstorage [Roadmap - Tooling and infrastructure]

changed title from [cea] Prepare a productized replaying stack to [CEA] Prepare a productized replaying stack

assigned to @vsellier

mentioned in commit swh-apps@90c605cf

mentioned in merge request swh-apps!33 (merged)

added platform::CEA label

mentioned in commit swh-apps@9b036853

mentioned in commit swh/infra/ci-cd/swh-charts@4f3610b4

mentioned in merge request swh/infra/ci-cd/swh-charts!278 (merged)

mentioned in commit swh/infra/ci-cd/swh-charts@415f07d1

mentioned in commit swh/infra/ci-cd/swh-charts@e015dbcb

mentioned in commit swh/infra/ci-cd/swh-charts@27fc3b7b

The deployment script is ready for review in swh/infra/ci-cd/swh-charts!278 (merged)

Some tasks are needed before landing the MR (if review is ok):

• Deploy the redis for the replayer error reportint (redis-winery-replay)
• Update the secret for kafka to add the user
• Create a sentry project for objstorage-replayer
• Add the sentry dsn in the MR

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@1fad8baa

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@c8a34f04

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@0e938cec

mentioned in commit swh/infra/ci-cd/swh-charts@5a70fea6

mentioned in commit swh/infra/ci-cd/swh-charts@3872f6eb

mentioned in commit swh/infra/ci-cd/swh-charts@14d9d339

mentioned in commit swh/infra/ci-cd/swh-charts@86289afd

The replayers are running in production.

5 replicas are configured.

The current traffic is around 12Mbits/s. I still not figure why it's not even closed of the numbers of #5193 (closed)

mentioned in commit swh/infra/ci-cd/swh-charts@edee4961

mentioned in commit swh/infra/ci-cd/swh-charts@9c873ab8

swh/devel/swh-objstorage-replayer!32 (merged) is not yet landed so the hashes stored in winery are sha1 which is not what is expected.

The packer failed:

DEBUG:swh.objstorage.backends.winery.throttler:20: sync others_count=1 total_usage=49
Traceback (most recent call last):
  File "/srv/winery/packer.py", line 14, in <module>
    result = objstorage.shard_packer(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/winery/venv-20231205.1/lib/python3.11/site-packages/swh/objstorage/backends/winery/objstorage.py", line 276, in shard_packer
    ret = pack(
          ^^^^^
  File "/srv/winery/venv-20231205.1/lib/python3.11/site-packages/swh/objstorage/backends/winery/objstorage.py", line 124, in pack
    ro.add(content, obj_id)
  File "/srv/winery/venv-20231205.1/lib/python3.11/site-packages/swh/objstorage/backends/winery/roshard.py", line 135, in add
    return self.throttler.throttle_add(self.shard.write, obj_id, content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/winery/venv-20231205.1/lib/python3.11/site-packages/swh/objstorage/backends/winery/throttler.py", line 203, in throttle_add
    return fun(obj_id, content)
           ^^^^^^^^^^^^^^^^^^^^
  File "/srv/winery/venv-20231205.1/lib/python3.11/site-packages/swh/perfecthash/__init__.py", line 134, in write
    raise ValueError(f"key length is {len(key)} instead of {Shard.key_len()}")
ValueError: key length is 20 instead of 32

This is now tagged as swh.objstorage.replayer v2.0.0

thanks

mentioned in commit swh-apps@4845a683

mentioned in commit swh/infra/ci-cd/swh-charts@c5e05176

mentioned in commit swh/infra/ci-cd/swh-charts@77b566ac

mentioned in commit swh-apps@0fa8d9b6

mentioned in commit swh/infra/ci-cd/swh-charts@b8d99efb

mentioned in commit swh-apps@88f4f2de

The changes are deployed in the production environment.

The ceph compression was also activated, the compression ratio should be 0.5 after some tests in the vagrant environment.

I will monitor everything during some time before closing the ticket

MR created in the EE repository: https://gitlab.easter-eggs.com/cst-she/ansible/-/merge_requests/10

mentioned in commit swh/infra/ci-cd/swh-charts@098c9742

The replaying stack is deployed in production and running since a couple of weeks.

closed