[cassandra] Replay the archive content

Monitor the archive content replay in cassandra

changed milestone to %Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]

added activity::Processing cassandra state:wip labels

assigned to @vsellier

added 1h of time spent at 2023-01-19

mentioned in commit swh/infra/ci-cd/swh-charts@e5226cb5

mentioned in commit swh/infra/ci-cd/swh-charts@aadc7798

mentioned in commit swh/infra/ci-cd/swh-charts@bca46cb9

mentioned in commit swh/infra/ci-cd/swh-charts@afde172e

mentioned in commit swh/infra/ci-cd/swh-charts@e5a33804

mentioned in commit swh/infra/ci-cd/swh-charts@09aa9ca9

Sadly we have plenty of corrupt objects without their raw_manifest (especially releases with missing signatures, and on staging where we didn't reinject bruteforced raw_manifests for past directories and revisions); so you should disable validation of replayers (validate: false in the config) because they don't match their checksum.

After discussing with @olasd, we'll ensure the objects are well stored on postgresql backend before adding them in cassandra.

To do so, we'll use the errors reported by the redis error reporter.

A first check is already implemented in errors-checker.py

The errors reported by the replayers are dispatched in several redis databases according to the effective error:

• db2: The content is found in kafka and postgresql and no hash validation is found, it's a real replaying problem, we can try to add the object in cassandra
• db3: The content is found in kafka but not in postgresql, the object can be ignored
• db4: The content is found in kafka, in postgresql but the hash validation failed, we'll add it to cassandra to avoid creating holes in the archive

To clearly identify the retry results, the impacted keys will be moved to distinct dbs when reinserted correctly or not:

• db2: ok -> db5 ; ko -> db6
• db4: ok -> db7 ; ko -> db8

The status is available in grafana at https://grafana.softwareheritage.org/d/xDLNRKUWz/redis-dashboard-kubernetes?orgId=1&refresh=30s&from=now-12h&to=now

• The storage should not have a journal entry configured to avoid re-adding the objects in kafka

the replaying of all the kafka topics is now fully complete. The replayers are following the new incoming objects.

\o/

Let's monitor and adapt the replayer autoscaling configuration to avoid too many pod count updates

closed