It's now able to log as json records.

Refs. swh/infra/sysadm-environment#4886

Refs. swh/infra/sysadm-environment#4886

Refs. swh/infra/sysadm-environment#4886

Refs. swh/infra/sysadm-environment#4886

Refs. swh/infra/sysadm-environment#4886

This requires rebuild of lister image to a more recent version with the pythonjsonlogger
dependency.

Refs. swh/infra/sysadm-environment#4886

Refs. swh/infra/sysadm-environment#4886

Related to swh/infra/sysadm-environment#4879

Refs. swh/infra/sysadm-environment#4883

[1] https://grafana.softwareheritage.org/goto/FVk2q9U4k?orgId=1

Refs. swh/infra/sysadm-environment#4883

[1] https://grafana.softwareheritage.org/goto/nLpc3rUVz?orgId=1

Refs. swh/infra/sysadm-environment#4883

[1] https://grafana.softwareheritage.org/goto/PkTD39UVz?orgId=1

Refs. swh/infra/sysadm-environment#4883

[1] https://grafana.softwareheritage.org/goto/PJnI3rUVk?orgId=1

Refs. swh/infra/sysadm-environment#4883

[1] https://grafana.softwareheritage.org/d/a164a7f0339f99e89cea5cb47e9be617/kubernetes-compute-resources-workload?orgId=1&var-datasource=Thanos&var-cluster=archive-staging-rke2&var-namespace=swh&var-workload=loader-bzr&var-type=deployment&from=now-24h&to=now

Refs. swh/infra/sysadm-environment#4883

Refs. swh/infra/sysadm-environment#4882

Deprecated to the benefit of savecodenow

Refs. swh/infra/sysadm-environment#4881

Refs. swh/infra/sysadm-environment#4881

Keeping the current behavior. A new image will be created with the right name so we can
drop the old highpriority reference after that.

Refs. swh/infra/sysadm-environment#4881

Refs. swh/infra/sysadm-environment#4881

This was historically named oneshot and it's no longer relevant.

We may create other "oneshot" queues but this will have to be for "oneshot"
workers (which no longer exists).

Refs. swh/infra/sysadm-environment#4873

It's currently wrongly 'changeme' in the index since it's been deployed with the
templatized chart.

Refs. swh/infra/sysadm-environment#4524

- {staging,production}-swh-logs
- {staging,production}-system-logs

This will fix the misnaming in production [1]. The data stream 'logs-*-*' pattern exists
which prevented the index name to be correctly built (and historically, the staging ones
were named differently too so the issue was not detected earlier).

[1]
```
curl -s ${ES_SERVER}/_cat/indices | grep "staging\|production"
...
green open  .ds-logs-swh-production-2023.05.16-000001    OMdrZ9DXRZ2t2SP4PEhwVA 1 1  5398438      0   2.1gb     1gb
green open  .ds-logs-system-production-2023.05.16-000001 WcV164JHSgOIOWjVUddeHg 1 1
6097451      0   3.4gb     2gb
...
```

Refs. swh/infra/sysadm-environment#4524

When no indexes key was present in the default values, we were forced to systematically
declare the index names (which was not initially intended). Adding this key make it
optional as intended (default values are computed in the chart).

Refs. swh/infra/sysadm-environment#4524

Refs. swh/infra/sysadm-environment#4524

When using 'limits', exceeding cpu requests is throttled. When pods exceeds memory
usage, the kube scheduler will kill the pod.

Using 'requests' instead will do nothing. It just allows the kube scheduler to determine
whether a pod can be scheduled or not (when pressure exists in the cluster).

We'll be monitoring through the dashboard to better determine the limits to configure
(per cluster).

[1] https://grafana.softwareheritage.org/goto/oPt0RLU4z?orgId=1

Refs. swh/infra/sysadm-environment#4524

Only 2 workers for production. It's still maximizing the cpu usage (~99%) and exceeding
the ram usage too [1] (max 14.68Gib).

[1] https://grafana.softwareheritage.org/goto/oPt0RLU4z?orgId=1

Refs. swh/infra/sysadm-environment#4524

Only 2 workers for production. It's maximizing the cpu usage (~99%) and exceeding the
ram usage too [1].

[1] https://grafana.softwareheritage.org/goto/90gX8fUVk?orgId=1

Refs. swh/infra/sysadm-environment#4524

Refs. swh/infra/sysadm-environment#4524

Refs. swh/infra/sysadm-environment#4880

Refs. swh/infra/sysadm-environment#4524

It makes the process gc every minute (or so) and lose logs.

Refs. swh/infra/sysadm-environment#4524

This fixes the missing version chart version, and the wrong chart to install.

Refs. swh/infra/sysadm-environment#4524

which may be the cause of the `ComparisonError: groupVersion shouldn't be empty`.
This is symptomatic of a missing apiVersion which is present but may be dropped due to
the helm chomping.

Refs. swh/infra/sysadm-environment#4524

Instead of the "default" namespace used so far.

Refs. swh/infra/sysadm-environment#4524

Refs. swh/infra/sysadm-environment#4524

Refs. swh/infra/sysadm-environment#4524

Dropping this version will deploy the same version as staging.

Refs. swh/infra/sysadm-environment#4878

Refs. swh/infra/sysadm-environment#4862