Refs. swh/infra/sysadm-environment#5226

Refs. swh/infra/sysadm-environment#5226

Refs. swh/infra/sysadm-environment#5215

As the intrinsic parallelism has increased, decrease the extrinsic
parallelism. ackLate allows us to disable "stopWhenNoActivity", and to
let autoscaling do its work.

Vincent Sellier authored 1 year ago and Nicolas Dandrimont committed 1 year ago

and remove unnecessary indirections

Antoine R. Dumont authored 1 year ago and Vincent Sellier committed 1 year ago

It seems to fail the same way the current storage rpc does.

Antoine R. Dumont authored 1 year ago and Vincent Sellier committed 1 year ago

That should avoid having cascading effect. When workers are too busy to handle that
probe, the http liveness probe fails, this ends up restarting the pod, in effect,
killing the ongoing requests.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-tcp-liveness-probe

Refs. swh/infra/sysadm-environment#5215

It's mandatory to allow prometheus to scrape the metrics directly from
the pod.

Related to swh/infra/sysadm-environment#5227

Limit the scraping to 1 pod as the metrics are the same for all pods

Related to swh/infra/sysadm-environment#5227

CPU isn't a great proxy for what we actually need (busy gunicorn
workers), but that's a start...

The space chopping was concatenating the yaml files together...

client_max_size is actually set at the toplevel of the swh configfile,
not within the objstorage.

The staging objstorage isn't especially fast

The network looks very slow this days. The consumers
fall in timeout and rafke ebalance all the consumers each time it appends.

Related to swh/infra/sysadm-environment#5187

Use 32 workers with 1 thread.

Refs. swh/infra/sysadm-environment#5215

For now, we don't know yet where it will settle and current 4 replicas does not follow
with half our workers.

Refs. swh/infra/sysadm-environment#5215

Refs. swh/infra/sysadm-environment#5215

Refs. swh/infra/sysadm-environment#5215

The last runs have finished so we can stop and decommission them.

Refs. swh/infra/sysadm-environment#5223

We migrated only around half the writers and the requests usage for both cpu and memory
is already reached.

Refs. swh/infra/sysadm-environment#5215

Refs. swh/infra/sysadm-environment#5215

Refs. swh/infra/sysadm-environment#5215

Refs. swh/infra/sysadm-environment#5215

The new one running on saam. Test it on one loader to incrementally check everything is
fine.

Refs. swh/infra/sysadm-environment#5215

This will soon be migrated.

Refs. swh/infra/sysadm-environment#5215

Refs. swh/infra/sysadm-environment#5215