[monitoring] Raise an alert when the number of messages in a rabbitmq queue is too high

We should be warned when the number of messages in a queue is reaching a given threshold.

Depends on #5036 (closed)

changed milestone to %MRO 2023

marked this issue as related to #4525

changed title from [monitorin] Raise an alert when the number of messages in a rabbitmq queue is too high to [monitoring] Raise an alert when the number of messages in a rabbitmq queue is too high

added Metrics/monitoring label

mentioned in commit swh/infra/ci-cd/swh-charts@a4e36956

mentioned in merge request swh/infra/ci-cd/swh-charts!162 (merged)

mentioned in commit swh/infra/ci-cd/swh-charts@9640ac44

mentioned in commit swh/infra/ci-cd/swh-charts@c86952ae

mentioned in commit swh/infra/ci-cd/swh-charts@eed2e2be

mentioned in commit swh/infra/ci-cd/swh-charts@4e0f6775

mentioned in commit swh/infra/ci-cd/swh-charts@fd4fbc56

mentioned in commit swh/infra/ci-cd/swh-charts@1328f59e

mentioned in commit swh/infra/ci-cd/swh-charts@22471e29

mentioned in commit swh/infra/ci-cd/swh-charts@186a4112

changed the description

It's been done [1], templatized and activated [2].

This led to a change in scraping metrics. Rabbitmq metrics were scrapped by the pergamon prometheus instance so it was not accessible by the cluster's alert manager (so we could not activate the alert mentioned in linked commits).

After making the metrics scraped by the cluster instead [2], the alert got activated. Impacted Grafana dashboards were also slightly adapted (they were using the old prometheus sources instead of Thanos).

And the prometheus pergamon scrapping of rabbitmq metrics were disabled on pergamon [3].

[1] swh/infra/ci-cd/k8s-clusters-conf!35 (closed) (successful test)

[2] swh/infra/ci-cd/swh-charts!333 (merged)

[3] swh/infra/puppet/puppet-swh-site!692 (merged)

closed

mentioned in issue #4525