Cassandra timeouts during vault on big repositories

This is just an issue to track and diagnose

Sentry Issue: SWH-STORAGE-2XA5

QueryTimeout: An unexpected error occurred in the api backend: ('Error from server: code=1200 [Coordinator node timed out waiting for replica nodes\' responses] message="Operation timed out - received only 0 responses." info={\'consistency\': \'LOCAL_QUORUM\', \'required_responses\': 2, \'received_responses\': 0}',)
(6 additional frame(s) were not displayed)
...
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/proxies/masking/__init__.py", line 386, in newf
    result = method(*args, **kwargs)
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/cassandra/storage.py", line 796, in directory_get_raw_manifest
    for dir_ in self._cql_runner.directory_get(directory_ids)
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/cassandra/cql.py", line 205, in newf
    return f(
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/cassandra/cql.py", line 749, in directory_get
    self._execute_with_retries(statement, [directory_ids]),
  File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/cassandra/cql.py", line 400, in _execute_with_retries
    raise QueryTimeout(*e.args) from None

It appends for for this revision for example: swh:1:rev:53396a22afc04536ddf75d8f82ad2eafa5082725

moved from swh/devel/swh-storage#4715 (moved)

changed the description

changed title from Cassandra timeouts during vault of big repositories to Cassandra timeouts during vault on big repositories

mentioned in issue #5507

marked this issue as related to #5527