Find a way to relabel the built-in rke2 node-exporter metrics to use the proper hostname of the nodes

The node-exporter metrics scraped from inside the k8s clusters are labeled with instance="$ip_address:$port" instead of the more meaningful instance=$fqdn label we've set on other hosts. This makes the associated alerts harder to track down; We should figure out a way to relabel these metrics.

added Metrics/monitoring label

The external metrics scraped use a relabeling entry definition [1] (the values are holding the intelligence there [2], upstream doc [3]).

Note though that i could not drop the :port part, it refused to work but still we have the <

$fqdn:$

port> in the metrics.

Maybe that'd help in this context too.

[3] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

[2] https://gitlab.softwareheritage.org/swh/infra/ci-cd/swh-charts/-/blob/production/cluster-components/values/archive-production-rke2.yaml?ref_type=heads#L27-40

[1] https://gitlab.softwareheritage.org/swh/infra/ci-cd/swh-charts/-/blob/production/cluster-components/templates/scrape-external-metrics/service-monitor.yaml?ref_type=heads#L26-28

assigned to @guillaume

mentioned in commit swh-sysadmin-provisioning@3598ab4b

mentioned in commit swh-sysadmin-provisioning@de92efe3

mentioned in commit swh-sysadmin-provisioning@0d31bc39

closed