ceph on proxmox slowness since 2024-02-29

Ceph on proxmox is slow since the upgrade of the ceph components to 17.2.7 on 2024-02-09.

marked this issue as related to #5280 (closed)

We've noticed a different memory usage pattern on the nodes: no OS cache is being used (for ceph OSDs, supposedly) anymore. https://grafana.softwareheritage.org/goto/KcfYzTAIk?orgId=1

We've tried to increase the OSD memory request, to no avail. https://grafana.softwareheritage.org/goto/7uAyDTAIz?orgId=1

Now that all critical k8s components are redunded, we should be able to issue a rolling reboot of the proxmox cluster.

The three proxmox ceph nodes have been rebooted

Moving all etcd traffic off of proxmox/ceph seems to have helped (?), we'll see...