Add loader-related metrics to swh-loader-core

Add some metrics to swh-loader-core

• fetch duration (histogram)
• number of objects of each kind added to the archive (counters)

---

Migrated from T1461 (view on Phabricator)

assigned to @olasd

marked this issue as related to swh/meta#1408 (closed)

added Core Loader Metrics/monitoring priority:High labels

unassigned @olasd

changed the description

marked the checklist item number of objects of each kind added to the archive (counters) as incomplete

I think the second point mostly happened: the storage is returning statistics to the loader, but the loaders don't generally collect them.

I'm not sure what kind of granularity we really want for the loader-level object counters (do we want per-loader breakdowns of objects added? per-visit?)

Once that's specified this is probably a candidate for an easy hack (as most of the data is there, it just needs to be collected / sent to statsd).

Closing this, as these metrics are now visible on https://grafana.softwareheritage.org/d/FqGC4zu7z/vlorentz-loader-metrics

There is still room for finer metrics though, such as swh-storage#4185

assigned to @vlorentz

closed