Design and implement tools to propagate removal to mirrors
We need to design and implement tools enabling mirrors to learn about removals and act on them if necessary. Desired outcome after disussing #4975 (closed):
- A dedicated topic is created in Kafka where the main archive publish processed take down requests with a reason for the request and a list of removed objects.
- A process on mirrors is subscribed to this Kafka topic and records these SWHID to a blocklist.
- This blocklist is used by
swh-storage
to prevent access to objects removed from the main archive. - An interface for operators (most probably on the web) allows to review the requests. For each request, operators can:
- propagate the request to their mirror and have objects deleted from their storage,
- deny the request and remove objects from the blocklist.
Edited by Jérémy Bobbio (Lunar)