Skip to content

[alertmanager] [Blocked] Update the configuration to be able to receive alert from other namespace than cattle-monitoring-system

Prometheus operator add by default a matcher on the main AlertManager(AM) rule to limit the alerts to the same namespace where the AlermanagerConfig is deployed.

In our case, the AMConfig is deployed on the cattle-monitoring-system so the alerts must have the label namespace=cattle-monitoring-system to match the routing.

 routes:
  - receiver: cattle-monitoring-system/irc-relay-config/ircrelay
    group_by:
    - '...'
    matchers:
    - namespace="cattle-monitoring-system"  <--- automagically added by the operator

An option to disable this behavior was added in the operator: https://github.com/prometheus-operator/prometheus-operator/pull/5084 and release in version v0.61.0

Unfortunately, we are using the rancher-monitoring helm chart to deploy the monitoring stack and the v102.0.1 version is highly coupled with the v0.59.1 version of the operator.

We should wait for a new release of the rancher-monitoring application to be able to disable this behavior.

In the meantime, we can configure our alerts to force the namespace: example

but we lose all the default alerts declared with the monitoring stack