[cassandra] holes in the monitoring

Prometheus is regularly missing cassandra metrics during a certain amount of time:

The problem is the same in production and staging, so it seems its a configuration issue or some kind of timeout somewhere

changed milestone to %Cassandra in production as primary storage [Roadmap - Tooling and infrastructure]

added Metrics/monitoring cassandra labels

assigned to @vsellier

or some long-standing process triggered regularly with a sufficiently high load preventing metrics from being gathered (e.g. gc) ?

indeed, it could be something like that.

Only the jmx metrics seem impacted. The point is all the metrics of all the servers are lost during the black hole which makes me think it could also have something not working correctly on prometheus or prometheus agents or cassandra service definition or ... ;)

the number of metrics exposed by cassandra is quite huge:

vsellier@cassandra01 ~ % curl -s http://localhost:7070 | grep -v "^#" | wc -l
24566

The default scape_timeout of a service is 10s.

- job_name: serviceMonitor/cassandra/cassandra-jmx-exporter/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http

the last scrapings of the metrics took between 4s and 6s, it's possible it reaches the timeout duration at some point. (from [1])

During my investigation, the target serviceMonitor/cassandra/cassandra-jmx-exporter/0 sometimes completely disappears from the targets.

There are no prometheus logs or kubernetes events explaining that.

[1] https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-75xcg59s/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-prometheus:9090/proxy/targets)

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@46ee60ac

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@472b2301

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@8cd52754

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@646b76de

The recommended way to do the manual endpoint declaration is now to use an Endpointslice object, but unfortunately, it seems prometheus doen't discover the cassandra nodes in this case: https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors

The point could be this kind of service should not declare any selector to not trigger the kubernetes auto discovery. The last commit is trying that, let's see if it improves something

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@25d45132

The monitoring is now stable without the service selector.

The root cause of the flip-flop seems to be the kubelet / api manager regularly restarting on both the staging and production cluster.
I will create another issue to address this problem.

closed

added 1h of time spent at 2023-05-10

added 2h of time spent at 2023-05-08

mentioned in issue #4876 (closed)