Skip to content
Snippets Groups Projects

Add a few statsd metrics in the kafka journal client

namely add 2 metrics:

  • swh_journal_client_handle_message_total: (counter) the total number of kafka messages that have been handled by the client,
  • swh_journal_client_status: (gauge) report the current status of the kafka consumer (waiting or processing).

Since these metrics will be created by any kafka consumer (any swh.journal.client), tags should be added to the statsd metrics (eg. via the STATS_TAGS env var) to distinguish between the different types of consumers (e.g. hostname, consumer type/role, etc.)

Especially for the swh_journal_client_status to work properly in an elastic execution environment (docker swarm, k8s, ...) it is mandatory to add a consumer-specific tag (e.g. the hostname) and make sure the TTL for the metrics in prometherus-statsd-exporter is pretty short (e.g. 1h), so that restarted consumers are not kept for too long/forever by statsd-exporter.

Depends on !213 (closed)


Migrated from D6944 (view on Phabricator)

Merge request reports

Approval is optional
Ready to merge by members who can write to the target branch.

Merge details

  • 6 commits will be added to master.
  • Source branch will not be deleted.

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading