Update scheduler metrics routine is taking too long

It triggered yesterday and 12h later [1] [2] it's not finished. It's actually supposed to trigger every 4h and used to take around ~20min.

That is blocking since the archive.softwareheritage.org's landing page is relying on it to display coverage statistics.

[1] status of the service triggering the update

● swh-scheduler-update-metrics.service - Software Heritage scheduler update-metrics
     Loaded: loaded (/etc/systemd/system/swh-scheduler-update-metrics.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-12-08 22:11:01 UTC; 12h ago

[2] pg_activity snapshot view at the time of the analysis

102070 softwar...eduler                      swhscheduler 192.168.100.210/   42.6  1.2   22.10M       0B 752:52.12  N    N             active   SELECT last_update, lister_id, origins_enabled, origins_known, origins_never_visited, ...

Migrated from T3785 (view on Phabricator)

Edited Oct 18, 2022 by Antoine R. Dumont