Activate gunicorn's statsd monitoring for objstorage services

https://docs.gunicorn.org/en/stable/instrumentation.html

on uffizi (and other?)

Related rSPSITE97742fb681f1741c0cfca9aeec01403c19d74bee

---

Migrated from T1592 (view on Phabricator)

added System administration label

changed title from Actiave gunicorn's statsd monitoring for objstorage services to Activate gunicorn's statsd monitoring for objstorage services

added priority:High label

Status:

Tryout with douardda and I failed so far. After setting the right logger in the configuration, it actually started logging but only workers event (gunicorn.workers: number of workers managed by the arbiter - gauge). Tried to dig in in the code but did not find the right solution yet.

There are apparently compatibility issues with the gunicorn workers we use (aiohttp) and gunicorn to send the information we are interested in (gunicorn.requests*). Well, at least with a default setup [1]

This still needs further investigation.

• [1] https://github.com/benoitc/gunicorn/issues/1300 https://github.com/aio-libs/aiohttp/issues/705 https://github.com/aio-libs/aiohttp/pull/1117

changed the description

Something is fishy somewhere. We tried [1] and did not notice that behavior.

https://docs.aiohttp.org/en/stable/logging.html#gunicorn-access-logs

And to be thorough, @douardda checked back on his machine and the behavior is correct there (even with the same version as the production).

mentioned in commit swh-apps@c6b66022

mentioned in merge request swh-apps!41 (merged)

mentioned in commit swh-apps@871ba4db

mentioned in commit swh-apps@b4569299

mentioned in commit swh/infra/ci-cd/swh-charts@9001944c

mentioned in commit swh/infra/ci-cd/swh-charts@96472bcd

mentioned in commit swh/infra/ci-cd/swh-charts@95a56ee5

mentioned in commit swh/infra/ci-cd/swh-charts@be3c56b3

mentioned in merge request swh/infra/ci-cd/swh-charts!453 (closed)

mentioned in commit swh/infra/ci-cd/swh-charts@29474304

I spent some time trying to check whether we could activate those metrics (focusing on the objstorage for now).

I had to evolve the objstorage image so we could activate the statsd relative flags [1].

Then I had to also evolve the objstorage deployment to expose another environment variable to ease the objstorage metric configuration [2]. The end goal being able to map those metrics with statsd_exporter [2]

I've been fighting the mapping as some documented metrics [3] are not outputted/mapped. But statsd exporter is not raising any issues with the mapping nor the debug level being helpful at all.

One of the missing metrics though requests_workers would be interesting to have for the autoscaling part (as in this example [4]).

Besides that, that sounds ok.

I'll stop there for now.

Note that i'm not sure whether I have some other keys to add (like the environment) or whether it's our overall monitoring orchestration that will do it.

[1] swh-apps!41 (merged)

[2] swh/infra/ci-cd/swh-charts!453 (closed)

[3] https://docs.gunicorn.org/en/stable/instrumentation.html

[4] https://medium.com/@sinaan71/kubernetes-hpa-based-on-statsd-metrics-monitoring-ee6178d9c339

I'll stop there for now.

or not.

Note that i'm not sure whether I have some other keys to add (like the environment) or whether it's our overall monitoring orchestration that will do it.

Well, to ensure that, the next step would be to run the test in staging. I've adapted the attached mr to that end (only testing it in staging).

I've been fighting the mapping as some documented metrics [3] are not outputted/mapped. But statsd exporter is not raising any issues with the mapping nor the debug level being helpful at all. One of the missing metrics though requests_workers would be interesting to have for the autoscaling part (as in this example [4]).

d'oh, the objstorage pods needed to be restarted for those to appear.

curl -s http://localhost:9102/metrics | grep gunicorn_worker
# HELP objstorage_read_only_gunicorn_workers Metric autogenerated by statsd_exporter.
# TYPE objstorage_read_only_gunicorn_workers gauge
objstorage_read_only_gunicorn_workers 2
# HELP objstorage_read_write_gunicorn_workers Metric autogenerated by statsd_exporter.
# TYPE objstorage_read_write_gunicorn_workers gauge
objstorage_read_write_gunicorn_workers 2
tony@kessel:~/work/swh/sysadm-environment/swh-charts (add-objstorage-gunicorn-statsd) $ curl -s http://localhost:9102/metrics | grep gunicorn_log
# HELP gunicorn_log Metric autogenerated by statsd_exporter.
# TYPE gunicorn_log counter
gunicorn_log{deployment="objstorage-read-only",level="error"} 4
gunicorn_log{deployment="objstorage-read-write",level="error"} 4

added 1d 4h of time spent

mentioned in commit swh/infra/ci-cd/swh-charts@e0bf6ace