Inconsistent version of swh-storage on prod loaders

Prod loaders have version 1.7.0 of swh-storage (which doesn't have swh.storage.interface.HashDict), but somehow a newer version of swh/storage/proxies/filter.py (which tries to import swh.storage.interface.HashDict), causing crashes in the last two hours.

Sentry Issue: SWH-LOADER-GIT-1HR

ImportError: cannot import name 'HashDict' from 'swh.storage.interface' (/usr/lib/python3/dist-packages/swh/storage/interface.py)
(11 additional frame(s) were not displayed)
...
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "swh/storage/proxies/filter.py", line 18, in <module>
    from swh.storage.interface import HashDict, StorageInterface

changed title from Inconsistent version of swh-storage on prod loader to Inconsistent version of swh-storage on prod loaders

Sentry issue: SWH-LOADER-SVN-E4

Sentry issue: SWH-LOADER-CORE-1G6

Sentry issue: SWH-LOADER-MERCURIAL-30

That's fixed already.

mentioned in issue #4806 (closed)

closed

Sentry disagrees, it's still happening on the PyPI and Mercurial loaders

reopened

Thanks for the ping.

I had other workers service to still restart (which got through last time).

That issue is really annoying... That change in the storage interface got completely misjudge in its impact (client side). Next time, if we are still on the static infra for the workers, I'll just plainly restart the workers after the upgrade (we just have too many swh-worker@$stuff.service to manually handle).

Like i said somewhere else, that should no longer happen with pods as their runtime dependencies are rebuilt as one set (so it's all consistent there upon successful deployement and restart).

I'm sure it's ok now [1] as nothing more is getting through this check command (where as it was the case before).

[1]

root@pergamon:~# clush -w @swh-workers systemctl status 'swh-worker@loader_*' | grep -i 'raised unexpected'

closed

added 2h of time spent

Sentry says it happened again, on the deposit loader this time

reopened

Sentry issue: SWH-LOADER-CORE-1HA

service swh-worker@(checker|loader)_deposit restarted.

closed