Skip to content

Add a few statsd metrics in the kafka journal client

David Douard requested to merge generated-differential-D6944-source into generated-differential-D6944-target

namely add 2 metrics:

  • swh_journal_client_handle_message_total: (counter) the total number of kafka messages that have been handled by the client,
  • swh_journal_client_status: (gauge) report the current status of the kafka consumer (waiting or processing).

Since these metrics will be created by any kafka consumer (any swh.journal.client), tags should be added to the statsd metrics (eg. via the STATS_TAGS env var) to distinguish between the different types of consumers (e.g. hostname, consumer type/role, etc.)

Especially for the swh_journal_client_status to work properly in an elastic execution environment (docker swarm, k8s, ...) it is mandatory to add a consumer-specific tag (e.g. the hostname) and make sure the TTL for the metrics in prometherus-statsd-exporter is pretty short (e.g. 1h), so that restarted consumers are not kept for too long/forever by statsd-exporter.

Depends on !213 (closed)

Migrated from D6944 (view on Phabricator)

Merge request reports