Validate data replayed in winery

changed milestone to %Scale-out objstorage in production as primary objstorage [Roadmap - Tooling and infrastructure]

mentioned in commit swh/devel/snippets@f00cfc1b

extract the replayer current positions per partition:

 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --describe --group swh-archive-prod-winery-content-replayer |grep -v PARTITION | awk '{print $3,$4}' | sort -n
0 3857735
1 5088832
2 4253136
3 4071316
4 3694360
...
252 4651455
253 5356356
254 5669851
255 4231367

A a first initial try to check the behavior, it was launch from the production kubernetes cluster, it seems it tooks ~2 days to check a partition:

partition=0 object_checked=25212        not_found=0     incorrect_hash=0        to_check=3288889 time_to_completion=1 day, 22:00:00.589842      (24.97/s)
partition=0 object_checked=25229        not_found=0     incorrect_hash=0        to_check=3288872 time_to_completion=1 day, 22:00:18.669682      (16.98/s)
partition=0 object_checked=25256        not_found=0     incorrect_hash=0        to_check=3288845 time_to_completion=1 day, 21:59:30.632848      (26.97/s)

The script still need to be updated to save the swhid of the objects in error.

This first attempt was done using the haproxy which is certainly not the optimal way to do.

The script could be splitted to serialized the ids to check in a file in a first part, which can be launched at rocquencourt and a second part could reuse the list of ids to directly request winery on the cea part.