Admin db node (dali) out of disk space (again)

Some zfs snapshots were cleaned up to free space. It's still growing fast due to sentry being a disk resource glouton.

• We decided to double its storage disk to 350G.

Another idea thrown in the air would be to send more snapshots to the backup machine and clean up the local snapshots (but the last one which is needed for zfs to continue working).

Refs. #4695 (closed)

changed milestone to %MRO 2023

added activity::Processing label

changed title from Dali out of disk space (again) to Admin db node (dali) out of disk space (again)

changed the description

looks like we have a peak of sentry messages since the 2023-03-20:

sentry=# select extract(day from timestamp), extract(month from timestamp), count(id) from nodestore_node group by 1,2 order by 3 desc limit 10;
 extract | extract |  count
---------+---------+---------
      20 |       3 | 1317377
      21 |       3 | 1016043
      13 |       1 |  785877
      14 |       3 |  473148
      22 |       3 |  423799
      15 |       3 |  364067
      19 |       1 |  359613
      13 |       3 |  327351
      14 |       2 |  327109
      28 |       1 |  312390
(10 rows)

Looks like https://sentry.softwareheritage.org/share/issue/0b4085ee918446bea20665cf5a359d7f/ sends a lot of fairly heavy (40kB) events

cross link to #4745

looks like the saam's replayers are encountering some problems since the 2023-03-20

From proxmox-ui, i've bumped the size to 350G.

Which is detected in the vm :

root@dali:~# date
Wed 22 Mar 2023 02:06:10 PM UTC
root@dalie:~# dmesg --follow --human
...
[  +0.000019] vdb: detected capacity change from 268435456000 to 375809638400
root@dali:~# parted /dev/vdb
exit
# And then it auto-expanded the disk... it probably would have done it automatically without any intervention either
[Mar22 14:03] virtio_blk virtio2: [vdb] new size: 734003200 512-byte logical blocks (376 GB/350 GiB)
[  +0.000019] vdb: detected capacity change from 268435456000 to 375809638400

mentioned in issue #4817 (closed)

changed the description

mentioned in commit swh-sysadmin-provisioning@8801a502

Oh and now, we have some space left:

root@dali:~# df -h /srv/postgresql/14/main
Filesystem       Size  Used Avail Use% Mounted on
data/postgresql  283G  177G  106G  63% /srv/postgresql/14/main

changed the description

Another issue to investigate the replayer issue is opened [1]

[1] #4817 (closed)

closed

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@967178ac

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@a23ab6db

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@e478ee95

mentioned in commit swh/infra/ci-cd/swh-charts@6eb5198a