It must be initialized manually for the moment

Related to swh/infra/sysadm-environment#5391

It's mostly probable that local clusters don't have a prometheus
operator deployed.

The refresh_history endpoint takes more than 10s to
retrieve the data from prometheus and compute the values.
The cronjob regularly failed probably until enough
data is in the thanos cache.

Related to swh/infra/sysadm-environment#5387

Related to swh/infra/sysadm-environment#5386

- Use the retry command line
- Split the rpc checks from the refresh endpoint call
- Fix a couple of minor side issues

Related to swh/infra/sysadm-environment#5387

Same reason as for the production

The number of requests sent to the ingress is known and limited
so the number of webapp can be fined managed.

It allows to remove all the autoscaling configuration.

Ref. swh/infra/sysadm-environment#5369

Related to swh/infra/sysadm-environment#5386

Use only 5XX errors to detect errors ratio.
Using the 4XX status can trigger false positive alerting
for some incomplete object storage or the webapp when
quota are exceeded

As there is now only 500 errors, the threshold can be reduced.
During the last month, it seems a the 10% error rate was
reached when real incident happened.

Use a dedicated ingress to use a regexp.
It adds more flexibility on the static resource management

Related to swh/infra/sysadm-environment#5383

Refs. swh/infra/sysadm-environment#5379

Refs. swh/infra/sysadm-environment#5379

Even if the cpu is not a good metrics, reducing this
ratio allow to start more replicas of the read-only storage.
It seems the webapp slowness are somehow due to slowness
of the read-only storage. More details are in the linked
issue.

swh/infra/sysadm-environment#5373

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327

Refs. swh/infra/sysadm-environment#5327