• Lead: vlorentz
• Priority: medium
• Effort: ??

Description:

Set up background jobs to regularly check - and repair when necessary - data validity, in all SWH data stores. This includes both blobs (swh-objstorage) and other graph objects (swh-storage) on all the copies (in-house, kafka, azure, upcoming mirrors, etc.)

Includes work:

• Implement storage scrubber for Cassandra
• Add scrubbing for the object storage
• Add metrics and Grafana dashboard for scrubbing process
• Automatically repair and recover objects found to be invalid

KPIs:

• List of scrubbers deployed in production
• Monitoring tools deployed in production
• Rolling report of operations per datastore including errors found and fixed at each iteration