Timeouts on the cassandra storage on the directory_ls method
Sentry Issue: SWH-INDEXER-SG
With version of swh-indexer v3.2.0
RemoteException: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
(25 additional frame(s) were not displayed)
...
File "swh", line 8, in <module>
sys.exit(main())WARNING:swh.storage.proxies.retry:Retrying RPC call │
Traceback (most recent call last): │
File "/opt/swh/.local/bin/swh", line 8, in <module> │
sys.exit(main()) │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main │
return swh(auto_envvar_prefix="SWH") │
File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ │
return self.main(*args, **kwargs) │
File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main │
rv = self.invoke(ctx) │
File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke │
return _process_result(sub_ctx.command.invoke(sub_ctx)) │
File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke │
return _process_result(sub_ctx.command.invoke(sub_ctx)) │
File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke │
return ctx.invoke(self.callback, **ctx.params) │
File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke │
return __callback(*args, **kwargs) │
File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func │
return f(get_current_context(), *args, **kwargs) │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/cli.py", line 280, in journal_client │
client.process(worker_fn) │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/journal/client.py", line 405, in process │
batch_processed, at_eof = self.handle_messages(messages, worker_fn) │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/journal/client.py", line 444, in handle_messages │
worker_fn(dict(objects)) │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/cli.py", line 277, in worker_fn │
fn(objects) │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/indexer.py", line 442, in process_journal_objects │
results = self.index_list( │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/metadata.py", line 526, in index_list │
for dir_metadata in self.directory_metadata_indexer.index(directory_id): │
File "/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/metadata.py", line 326, in index │
list(self.storage.directory_ls(id, recursive=False)), │Activity
added cassandra environment: staging labels
The problem is also present in production and for the webapp.
With a storage directly connected on the staging cassandra server, the directory_ls takes around 3 minutes to reply for the directory
swh:1:dir:86691db0f3931a9bf6f4684cb7af67129ea02ad5(linked to the revisionswh:1:rev:a10ea171210f7bd1f08877194eae8d816323723c)For the postgresql, the statement is cancelled.
For the record, the directory has 26068 entries.
Interesting fact: The response time in production is around 1min30
Edited by Vincent Sellieradded environment: production label
Please register or sign in to reply