staging/gunicorn-objstorage: Activate statsd metrics
This aims at activating the metrics for gunicorn. For now, it's a test on the objstorage instance running in staging.
A test has been running locally (lifting the local-cluster with monitoring activated \o/). The instance runs 2 objstorage instances (rw, ro). After applying the following and restarting the pods, we obtain the following metrics [1]
metrics sample extract
$ curl -s http://localhost:9102/metrics | grep gunicorn_
# HELP gunicorn_log Metric autogenerated by statsd_exporter.
# TYPE gunicorn_log counter
gunicorn_log{deployment="objstorage-read-only",level="error"} 4
gunicorn_log{deployment="objstorage-read-write",level="error"} 4
# HELP gunicorn_request_duration Metric autogenerated by statsd_exporter.
# TYPE gunicorn_request_duration histogram
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.005"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.01"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.025"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.05"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.1"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.25"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.5"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="0.75"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="1"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="2"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="5"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="10"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="15"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="30"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="45"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="60"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="120"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="300"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="600"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="900"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="1800"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="2700"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="3600"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="7200"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-only",le="+Inf"} 313
gunicorn_request_duration_sum{deployment="objstorage-read-only"} 0.255027
gunicorn_request_duration_count{deployment="objstorage-read-only"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.005"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.01"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.025"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.05"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.1"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.25"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.5"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="0.75"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="1"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="2"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="5"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="10"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="15"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="30"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="45"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="60"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="120"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="300"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="600"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="900"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="1800"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="2700"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="3600"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="7200"} 313
gunicorn_request_duration_bucket{deployment="objstorage-read-write",le="+Inf"} 313
gunicorn_request_duration_sum{deployment="objstorage-read-write"} 0.31060600000000044
gunicorn_request_duration_count{deployment="objstorage-read-write"} 313
# HELP gunicorn_requests Metric autogenerated by statsd_exporter.
# TYPE gunicorn_requests counter
gunicorn_requests{deployment="objstorage-read-only"} 313
gunicorn_requests{deployment="objstorage-read-write"} 313
# HELP gunicorn_response_code gunicorn response code
# TYPE gunicorn_response_code counter
gunicorn_response_code{deployment="objstorage-read-only",status="200"} 313
gunicorn_response_code{deployment="objstorage-read-write",status="200"} 313
# HELP gunicorn_workers Metric autogenerated by statsd_exporter.
# TYPE gunicorn_workers gauge
gunicorn_workers{deployment="objstorage-read-only"} 2
gunicorn_workers{deployment="objstorage-read-write"} 2helm diff
[swh] Comparing changes between branches production and add-objstorage-gunicorn-statsd (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in add-objstorage-gunicorn-statsd branch for environment staging...
[swh] Generate config in add-objstorage-gunicorn-statsd branch for environment staging...
[swh] Generate config in add-objstorage-gunicorn-statsd branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in add-objstorage-gunicorn-statsd branch for environment production...
[swh] Generate config in add-objstorage-gunicorn-statsd branch for environment production...
[swh] Generate config in add-objstorage-gunicorn-statsd branch for environment production...
------------- diff for environment staging namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.6YSE2kJg/staging-swh.before, 135 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.6YSE2kJg/staging-swh.after, 135 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned three differences
|___/
data.config.yml (v1/ConfigMap/swh/prometheus-statsd-exporter)
± value change in multiline text (one insert, no deletions)
+ - match: "*.gunicorn.log.*"
+ name: "gunicorn_log"
+ labels:
+ deployment: "$1"
+ level: "$2"
+ - match: "*.gunicorn.request.status.*"
+ help: "gunicorn response code"
+ name: "gunicorn_response_code"
+ labels:
+ deployment: "$1"
+ status: "$2"
+ - match: "*.gunicorn.requests"
+ name: "gunicorn_requests"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.workers"
+ name: "gunicorn_workers"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.request.duration"
+ name: "gunicorn_request_duration"
+ labels:
+ deployment: "$1"
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh/prometheus-statsd-exporter)
± value change
- 7b6e0c1b06b618203d65c94826eb903ab3765f3fb3f6f031913d8b2d4bdf9517
+ f5fc089902c619aedf70ec8d0737e3ec75e274b985568e3594369d3e98c3b1f2
spec.template.spec.containers.prometheus-statsd-exporter.args (apps/v1/Deployment/swh/prometheus-statsd-exporter)
+ one list entry added:
- "--log.level=info"
------------- diff for environment staging namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.6YSE2kJg/staging-swh-cassandra.before, 415 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.6YSE2kJg/staging-swh-cassandra.after, 415 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned 13 differences
|___/
data.config.yml (v1/ConfigMap/swh-cassandra/prometheus-statsd-exporter)
± value change in multiline text (one insert, no deletions)
+ - match: "*.gunicorn.log.*"
+ name: "gunicorn_log"
+ labels:
+ deployment: "$1"
+ level: "$2"
+ - match: "*.gunicorn.request.status.*"
+ help: "gunicorn response code"
+ name: "gunicorn_response_code"
+ labels:
+ deployment: "$1"
+ status: "$2"
+ - match: "*.gunicorn.requests"
+ name: "gunicorn_requests"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.workers"
+ name: "gunicorn_workers"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.request.duration"
+ name: "gunicorn_request_duration"
+ labels:
+ deployment: "$1"
spec.template.spec.containers.objstorage-db1-read-only.image (apps/v1/Deployment/swh-cassandra/objstorage-db1-read-only)
± value change
- container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20240702.1
+ container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage-statsd-test:20240807.3
spec.template.spec.containers.objstorage-db1-read-only.env (apps/v1/Deployment/swh-cassandra/objstorage-db1-read-only)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-db1-read-only
spec.template.spec.containers.objstorage-db1-read-write.image (apps/v1/Deployment/swh-cassandra/objstorage-db1-read-write)
± value change
- container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20240702.1
+ container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage-statsd-test:20240807.3
spec.template.spec.containers.objstorage-db1-read-write.env (apps/v1/Deployment/swh-cassandra/objstorage-db1-read-write)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-db1-read-write
spec.template.spec.containers.objstorage-multiplexer-read-only.image (apps/v1/Deployment/swh-cassandra/objstorage-multiplexer-read-only)
± value change
- container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20240702.1
+ container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage-statsd-test:20240807.3
spec.template.spec.containers.objstorage-multiplexer-read-only.env (apps/v1/Deployment/swh-cassandra/objstorage-multiplexer-read-only)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-multiplexer-read-only
spec.template.spec.containers.objstorage-multiplexer-read-write.image (apps/v1/Deployment/swh-cassandra/objstorage-multiplexer-read-write)
± value change
- container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20240702.1
+ container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage-statsd-test:20240807.3
spec.template.spec.containers.objstorage-multiplexer-read-write.env (apps/v1/Deployment/swh-cassandra/objstorage-multiplexer-read-write)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-multiplexer-read-write
spec.template.spec.containers.objstorage-storage1-read-only.image (apps/v1/Deployment/swh-cassandra/objstorage-storage1-read-only)
± value change
- container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20240702.1
+ container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage-statsd-test:20240807.3
spec.template.spec.containers.objstorage-storage1-read-only.env (apps/v1/Deployment/swh-cassandra/objstorage-storage1-read-only)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-storage1-read-only
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra/prometheus-statsd-exporter)
± value change
- acf4e3b78966c7df9981a846a50918d941354bb0285876e3ffadb86ab14a040c
+ 635f1a9fadf9cf5a40adeeff5b0def7e6f8e2c507a6745c128e4b57cc6d94a0e
spec.template.spec.containers.prometheus-statsd-exporter.args (apps/v1/Deployment/swh-cassandra/prometheus-statsd-exporter)
+ one list entry added:
- "--log.level=info"
------------- diff for environment staging namespace swh-cassandra-next-version -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.6YSE2kJg/staging-swh-cassandra-next-version.before, 319 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.6YSE2kJg/staging-swh-cassandra-next-version.after, 319 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned five differences
|___/
data.config.yml (v1/ConfigMap/swh-cassandra-next-version/prometheus-statsd-exporter)
± value change in multiline text (one insert, no deletions)
+ - match: "*.gunicorn.log.*"
+ name: "gunicorn_log"
+ labels:
+ deployment: "$1"
+ level: "$2"
+ - match: "*.gunicorn.request.status.*"
+ help: "gunicorn response code"
+ name: "gunicorn_response_code"
+ labels:
+ deployment: "$1"
+ status: "$2"
+ - match: "*.gunicorn.requests"
+ name: "gunicorn_requests"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.workers"
+ name: "gunicorn_workers"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.request.duration"
+ name: "gunicorn_request_duration"
+ labels:
+ deployment: "$1"
spec.template.spec.containers.objstorage-rw-pathslicing.image (apps/v1/Deployment/swh-cassandra-next-version/objstorage-rw-pathslicing)
± value change
- container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage:20240702.1
+ container-registry.softwareheritage.org/swh/infra/swh-apps/objstorage-statsd-test:20240807.3
spec.template.spec.containers.objstorage-rw-pathslicing.env (apps/v1/Deployment/swh-cassandra-next-version/objstorage-rw-pathslicing)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-rw-pathslicing
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra-next-version/prometheus-statsd-exporter)
± value change
- 85526c8da1e7109721abd43864853fc23985e22cc09e582269b8ef402585be71
+ 3f321aa8e3772ebc849114ee8f4159220be4ee64bd27083e66942635b5b4087e
spec.template.spec.containers.prometheus-statsd-exporter.args (apps/v1/Deployment/swh-cassandra-next-version/prometheus-statsd-exporter)
+ one list entry added:
- "--log.level=info"
------------- diff for environment production namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.6YSE2kJg/production-swh.before, 426 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.6YSE2kJg/production-swh.after, 426 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned six differences
|___/
data.config.yml (v1/ConfigMap/swh/prometheus-statsd-exporter)
± value change in multiline text (one insert, no deletions)
+ - match: "*.gunicorn.log.*"
+ name: "gunicorn_log"
+ labels:
+ deployment: "$1"
+ level: "$2"
+ - match: "*.gunicorn.request.status.*"
+ help: "gunicorn response code"
+ name: "gunicorn_response_code"
+ labels:
+ deployment: "$1"
+ status: "$2"
+ - match: "*.gunicorn.requests"
+ name: "gunicorn_requests"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.workers"
+ name: "gunicorn_workers"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.request.duration"
+ name: "gunicorn_request_duration"
+ labels:
+ deployment: "$1"
spec.template.spec.containers.objstorage-read-only.env (apps/v1/Deployment/swh/objstorage-read-only)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-read-only
spec.template.spec.containers.objstorage-ro-banco-xfs.env (apps/v1/Deployment/swh/objstorage-ro-banco-xfs)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-ro-banco-xfs
spec.template.spec.containers.objstorage-ro-saam-zfs.env (apps/v1/Deployment/swh/objstorage-ro-saam-zfs)
+ one list entry added:
- name: STATSD_SERVICE_TYPE
value: objstorage-ro-saam-zfs
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh/prometheus-statsd-exporter)
± value change
- 7b6e0c1b06b618203d65c94826eb903ab3765f3fb3f6f031913d8b2d4bdf9517
+ f5fc089902c619aedf70ec8d0737e3ec75e274b985568e3594369d3e98c3b1f2
spec.template.spec.containers.prometheus-statsd-exporter.args (apps/v1/Deployment/swh/prometheus-statsd-exporter)
+ one list entry added:
- "--log.level=info"
------------- diff for environment production namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.6YSE2kJg/production-swh-cassandra.before, 122 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.6YSE2kJg/production-swh-cassandra.after, 122 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned three differences
|___/
data.config.yml (v1/ConfigMap/swh-cassandra/prometheus-statsd-exporter)
± value change in multiline text (one insert, no deletions)
+ - match: "*.gunicorn.log.*"
+ name: "gunicorn_log"
+ labels:
+ deployment: "$1"
+ level: "$2"
+ - match: "*.gunicorn.request.status.*"
+ help: "gunicorn response code"
+ name: "gunicorn_response_code"
+ labels:
+ deployment: "$1"
+ status: "$2"
+ - match: "*.gunicorn.requests"
+ name: "gunicorn_requests"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.workers"
+ name: "gunicorn_workers"
+ labels:
+ deployment: "$1"
+ - match: "*.gunicorn.request.duration"
+ name: "gunicorn_request_duration"
+ labels:
+ deployment: "$1"
spec.template.metadata.annotations.checksum/config (apps/v1/Deployment/swh-cassandra/prometheus-statsd-exporter)
± value change
- acf4e3b78966c7df9981a846a50918d941354bb0285876e3ffadb86ab14a040c
+ 635f1a9fadf9cf5a40adeeff5b0def7e6f8e2c507a6745c128e4b57cc6d94a0e
spec.template.spec.containers.prometheus-statsd-exporter.args (apps/v1/Deployment/swh-cassandra/prometheus-statsd-exporter)
+ one list entry added:
- "--log.level=info"Grafana tryouts with local-cluster environment running 2 objstorage instances.
Depends on swh/infra/swh-apps!41 (merged)
Edited by Antoine R. Dumont