components/alerting: Add KEDA scaledobjects alert
Related to swh/infra/sysadm-environment#5362 (closed)
These modifications will create an alert if a scaledobject shows errors within the last 15 minutes in staging and production environment.
Helm-diff
[cluster-components] Comparing changes between branches production and scaled_object_alert...
Your branch is up to date with 'origin/production'.
[cluster-components] Generate config in production branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/local-cluster.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in production branch for cluster-components/values/test-staging-rke2.yaml...
Your branch is up to date with 'origin/scaled_object_alert'.
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/admin-rke2.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/archive-production-rke2.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/archive-staging-rke2.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/default.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/gitlab-production.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/gitlab-staging.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/local-cluster.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/rancher.yaml...
[cluster-components] Generate config in scaled_object_alert branch for cluster-components/values/test-staging-rke2.yaml...
------------- diff for cluster-components/values/admin-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/admin-rke2.yaml.before, 32 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/admin-rke2.yaml.after, 32 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/archive-production-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/archive-production-rke2.yaml.before, 23 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/archive-production-rke2.yaml.after, 23 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned one difference
|___/
spec.groups.swh-production.rules.rules (monitoring.coreos.com/v1/PrometheusRule/cattle-monitoring-system/swh-production.rules)
+ one list entry added:
- alert: KEDA_Scaled_Object_Errors_In_Production
annotations:
description: "Scaled object {{ $labels.exported_namespace }}/{{ $labels.scaledObject }} have encountered {{ $value }} errors within the last 15 minutes."
summary: "KEDA scaled object {{ $labels.scaledObject }} in error."
expr: "sum by(scaledObject) (rate(keda_scaled_object_errors_total{exported_namespace!="swh-cassandra-next-version"}[15m])) > 0"
for: 15m
labels:
severity: warning
namespace: cattle-monitoring-system
------------- diff for cluster-components/values/archive-staging-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/archive-staging-rke2.yaml.before, 55 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/archive-staging-rke2.yaml.after, 55 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned one difference
|___/
spec.groups.swh-staging.rules.rules (monitoring.coreos.com/v1/PrometheusRule/cattle-monitoring-system/swh-staging.rules)
+ one list entry added:
- alert: KEDA_Scaled_Object_Errors_In_Staging
annotations:
description: "Scaled object {{ $labels.exported_namespace }}/{{ $labels.scaledObject }} have encountered {{ $value }} errors within the last 15 minutes."
summary: "KEDA scaled object {{ $labels.scaledObject }} in error."
expr: "sum by(scaledObject) (rate(keda_scaled_object_errors_total{exported_namespace!="swh-cassandra-next-version"}[15m])) > 0"
for: 15m
labels:
severity: warning
namespace: cattle-monitoring-system
------------- diff for cluster-components/values/default.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/default.yaml.before, six documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/default.yaml.after, six documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/gitlab-production.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-production.yaml.before, seven documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-production.yaml.after, seven documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/gitlab-staging.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-staging.yaml.before, seven documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/gitlab-staging.yaml.after, seven documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/local-cluster.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/local-cluster.yaml.before, eight documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/local-cluster.yaml.after, eight documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/rancher.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/rancher.yaml.before, seven documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/rancher.yaml.after, seven documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for cluster-components/values/test-staging-rke2.yaml -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.cluster-components.MXBdQbxx/test-staging-rke2.yaml.before, 18 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.cluster-components.MXBdQbxx/test-staging-rke2.yaml.after, 18 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
Check the rule files
~/_swh_src/sysadm-environment/swh-charts/cluster-components (scaled_object_alert ✔) ᐅ helm template -f values.yaml -f values/archive-production-rke2.yaml alerting . | \
grep "^[[:space:]]*groups:" -A 72 | promtool check rules
Checking standard input
SUCCESS: 7 rules found