components/alerting: Create an alert for unreachable cassandra nodes

Related to swh/infra/sysadm-environment#5026 (closed)

This template will deploy a PrometheusRule in staging and production to trigger an alert if a cassandra cluster node is unreachable.

ᐅ helm template -f values.yaml -f values/archive-production-rke2.yaml cassandra-alerting . | grep -A 12 groups 
  groups:
  - name: critical-cassandra-service.rules
    rules:
    - alert: Cassandra_Service_Degraded_In_Production
      annotations:
        description: "The {{ $labels.instance }} node is unreachable for more than 5 minutes. This node seems down."
        summary: "The {{ $labels.service }} is degraded. Please check the {{ $labels.instance }} status."
      expr: up{service="cassandra-servers-svc"} == 0
      for: 5m
      labels:
        severity: critical
        namespace: cattle-monitoring-system

ᐅ helm template -f values.yaml -f values/archive-production-rke2.yaml cassandra-alerting . | \
grep -A 12 groups | promtool check rules
Checking standard input
  SUCCESS: 1 rules found

assigned to @guillaume

Nice.

I'm wondering whether this be should attached to the cassandra templates (in the swh chart) instead?

Like we did for the scheduler [1]

@vsellier what do you think (as you already mentioned in [2])?

[1] !162 (merged)

[2] !162 (comment 150071)

approved this merge request

added 1 commit

• 1ab6c384 - components/alerting: Create an alert for unreachable cassandra nodes

Compare with previous version

approved this merge request

added 1 commit

• a871c9d3 - components/alerting: Create an alert for unrepaired table size

Compare with previous version

resolved all threads

wow, even nicer ;)

Great ;)

We may need to adapt the size and the severity.

right, we'll know more once this is deployed and run for some time first ;p

added 4 commits

• a871c9d3...58cf5d91 - 2 commits from branch production
• dcd233f9 - components/alerting: Create an alert for unreachable cassandra nodes
• bb2cb443 - components/alerting: Create an alert for unrepaired table size

Compare with previous version

merged