Storage metrics not refreshed
Since the 2022-03-23, it seems the statsd storage metrics are not updated anymore.
It could be a regression of 284a4ab3066956dc403c4ada700e8421990c0528 (released in 1.1.0) as the date match with the upgrade from version 1.0.0 to version 1.2.0 of the python3-swh.storage package.
Start-Date: 2022-03-23 17:01:23
Commandline: apt dist-upgrade
Requested-By: olasd (1001)
Install: linux-headers-5.10.0-0.bpo.12-common:amd64 (5.10.103-1~bpo10+1, automatic), linux-headers-5.10.0-0.bpo.12-amd64:amd64 (5.10.103-1~bpo10+1, automatic), li
nux-image-5.10.0-0.bpo.12-amd64:amd64 (5.10.103-1~bpo10+1, automatic)
Upgrade: linux-kbuild-5.10:amd64 (5.10.92-1~bpo10+1, 5.10.103-1~bpo10+1), linux-image-amd64:amd64 (5.10.70-1~bpo10+1, 5.10.103-1~bpo10+1), python3-swh.model:amd64
(4.4.0-1~swh1~bpo10+1, 6.0.0-1~swh1~bpo10+1), linux-headers-amd64:amd64 (5.10.70-1~bpo10+1, 5.10.103-1~bpo10+1), python3-swh.storage:amd64 (1.0.0-1~swh1~bpo10+1,
1.2.0-1~swh1~bpo10+1), python3-swh.core:amd64 (2.2.1-1~swh1~bpo10+1, 2.2.2-1~swh1~bpo10+1)
End-Date: 2022-03-23 17:03:12The monitoring stack seems to work correctly as some statistics are still updated. for example, the stats of the content replayer:
# HELP swh_content_replayer_retries_total Metric autogenerated by statsd_exporter.
# TYPE swh_content_replayer_retries_total counter
swh_content_replayer_retries_total{attempt="1",operation="copy"} 1.2108223e+07
-swh_content_replayer_retries_total{attempt="1",operation="get_object"} 3.20125529e+08
-swh_content_replayer_retries_total{attempt="1",operation="put_object"} 3.17732857e+08
+swh_content_replayer_retries_total{attempt="1",operation="get_object"} 3.20156286e+08
+swh_content_replayer_retries_total{attempt="1",operation="put_object"} 3.17763416e+08
swh_content_replayer_retries_total{attempt="2",operation="copy"} 950575
-swh_content_replayer_retries_total{attempt="2",operation="put_object"} 2.3881e+06
+swh_content_replayer_retries_total{attempt="2",operation="put_object"} 2.388296e+06
swh_content_replayer_retries_total{attempt="3",operation="copy"} 9220
swh_content_replayer_retries_total{attempt="3",operation="put_object"} 446The storage duration are still updated but not the operation counts:
# HELP swh_objstorage_request_duration_seconds_error_count Metric autogenerated by statsd_exporter.
# TYPE swh_objstorage_request_duration_seconds_error_count counter
swh_objstorage_request_duration_seconds_error_count{endpoint="get_bytes"} 1166
@@ -1154,33 +1154,33 @@
swh_storage_request_duration_seconds_bucket{endpoint="extid_get_from_target",le="+Inf"} 3.956518e+06
swh_storage_request_duration_seconds_sum{endpoint="extid_get_from_target"} 39741.5300646238
swh_storage_request_duration_seconds_count{endpoint="extid_get_from_target"} 3.956518e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.005"} 1.084838e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.01"} 1.084912e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.025"} 1.084965e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.05"} 1.084974e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.1"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.25"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.5"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.75"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="1"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="2"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="5"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="10"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="15"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="30"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="45"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="60"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="120"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="300"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="600"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="900"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="1800"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="2700"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="3600"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="7200"} 1.084975e+06
-swh_storage_request_duration_seconds_bucket{endpoint="index",le="+Inf"} 1.084975e+06
-swh_storage_request_duration_seconds_sum{endpoint="index"} 11.127004264484611
-swh_storage_request_duration_seconds_count{endpoint="index"} 1.084975e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.005"} 1.084857e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.01"} 1.084931e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.025"} 1.084984e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.05"} 1.084993e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.1"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.25"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.5"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="0.75"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="1"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="2"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="5"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="10"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="15"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="30"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="45"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="60"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="120"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="300"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="600"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="900"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="1800"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="2700"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="3600"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="7200"} 1.084994e+06
+swh_storage_request_duration_seconds_bucket{endpoint="index",le="+Inf"} 1.084994e+06
+swh_storage_request_duration_seconds_sum{endpoint="index"} 11.127209654639927
+swh_storage_request_duration_seconds_count{endpoint="index"} 1.084994e+06Migrated from T4117 (view on Phabricator)