Dali almost out of disk space
Dali is again almost out of free disk space.
/usr/sbin/zfs list -t all
NAME USED AVAIL REFER MOUNTPOINT
data 190G 2.56G 24K /data
data/postgresql 190G 2.56G 113G /srv/postgresql/14/main
data/postgresql@autosnap_2022-12-03_00:00:01_daily 2.68G - 114G -
data/postgresql@autosnap_2022-12-04_00:00:02_daily 1.45G - 114G -
data/postgresql@autosnap_2022-12-05_00:00:02_daily 1.87G - 114G -
data/postgresql@autosnap_2022-12-06_00:00:00_daily 1.47G - 114G -
data/postgresql@autosnap_2022-12-07_00:00:02_daily 9.21G - 113G -
data/postgresql@autosnap_2022-12-08_00:00:02_daily 2.88G - 113G -
data/postgresql@autosnap_2022-12-09_00:00:00_daily 6.97G - 113G -
data/postgresql/wal 323M 2.56G 35.6M /srv/postgresql/14/main/pg_wal
data/postgresql/wal@autosnap_2022-12-03_00:00:01_daily 22.8M - 46.4M -
data/postgresql/wal@autosnap_2022-12-04_00:00:02_daily 33.1M - 56.7M -
data/postgresql/wal@autosnap_2022-12-05_00:00:02_daily 24.4M - 48.0M -
data/postgresql/wal@autosnap_2022-12-06_00:00:00_daily 33.8M - 49.8M -
data/postgresql/wal@autosnap_2022-12-07_00:00:02_daily 67.2M - 83.1M -
data/postgresql/wal@autosnap_2022-12-08_00:00:02_daily 43.3M - 59.2M -
data/postgresql/wal@autosnap_2022-12-09_00:00:00_daily 39.4M - 55.3M -Activity
added priority:High state:wip labels
assigned to @vsellier
I tried to restore the sentry database on a different postgresql to evaluate the effect of a full vacuum of the table
nodestore_node.
It seems there is no much gains as the db is still 100Go after a restore instead of 125Go for dali.The table seems correctly cleaned:
sentry=# select timestamp from nodestore_node order by timestamp asc limit 1; timestamp ------------------------------- 2022-09-12 12:47:49.926367+00 (1 row)I didn't found any kind of circuit breakdown in sentry that could avoid the peaks in the db like that:
So I guess the only way to quiet this problem is to increase the volume size.
(from an @olasd) It could also be possible to test the s3 storage (in the admin minio).
It needs some tests as it doesn't seems really production grade ready.We also need to be improve the minio monitoring and be sure it can handle the sentry load.
An example of s3 storage extension https://github.com/ewdurbin/sentry-s3-nodestore The documentation page of the nodestore: https://develop.sentry.dev/services/nodestore/
we need to check if it can be easily integrated in the docker container and if it works correctly.
mentioned in commit swh-sysadmin-provisioning@389efa6a
Looks like moving to s3 storage is a subject by itself so in the meantime, I increased the dali's data disk space by 50Go to give it some fresh air:
root@dali:~# zfs list -t all NAME USED AVAIL REFER MOUNTPOINT data 179G 62.5G 24K /data data/postgresql 179G 62.5G 115G /srv/postgresql/14/main data/postgresql@autosnap_2022-12-19_00:00:00_daily 2.58G - 114G - data/postgresql@autosnap_2022-12-20_00:00:00_daily 1.23G - 115G - data/postgresql@autosnap_2022-12-21_00:00:00_daily 1.13G - 115G - data/postgresql@autosnap_2022-12-22_00:00:01_daily 2.99G - 115G - data/postgresql@autosnap_2022-12-24_00:00:00_daily 1.36G - 115G - data/postgresql@autosnap_2022-12-25_00:00:02_daily 1.15G - 115G - data/postgresql@autosnap_2022-12-26_00:00:01_daily 1.19G - 115G - data/postgresql/wal 280M 62.5G 54.7M /srv/postgresql/14/main/pg_wal data/postgresql/wal@autosnap_2022-12-20_00:00:00_daily 23.0M - 23.0M - data/postgresql/wal@autosnap_2022-12-21_00:00:00_daily 32.3M - 32.4M - data/postgresql/wal@autosnap_2022-12-22_00:00:01_daily 37.5M - 37.6M - data/postgresql/wal@autosnap_2022-12-23_00:00:01_daily 32.8M - 32.9M - data/postgresql/wal@autosnap_2022-12-24_00:00:00_daily 29.5M - 37.5M - data/postgresql/wal@autosnap_2022-12-25_00:00:02_daily 36.2M - 44.2M - data/postgresql/wal@autosnap_2022-12-26_00:00:01_daily 25.6M - 33.6M -I will also migrate the data disk of backup01 in azure from a 256Go P disk to a 512Go (S15 to S20). It will create an additional cost of 11$ per month.
backup01 disk resized:
root@backup01:~# parted /dev/sdb GNU Parted 3.4 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 550GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 275GB 275GB zfs zfs-06495ac79981d8e6 (parted) resizepart 1 End? [275GB]? 550GB (parted) print Model: Msft Virtual Disk (scsi) Disk /dev/sdb: 550GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 550GB 550GB zfs zfs-06495ac79981d8e6root@backup01:~# zfs list -t all NAME USED AVAIL REFER MOUNTPOINT data 201G 294G 96K none data/sync 201G 294G 96K none data/sync/dali 201G 294G 96K none data/sync/dali/postgresql 201G 294G 125G none data/sync/dali/postgresql@autosnap_2022-12-20_00:00:00_daily 4.57G - 125G - data/sync/dali/postgresql@autosnap_2022-12-21_00:00:00_daily 2.54G - 125G - data/sync/dali/postgresql@autosnap_2022-12-22_00:00:01_daily 4.54G - 125G - data/sync/dali/postgresql@autosnap_2022-12-23_00:00:01_daily 5.11G - 125G - data/sync/dali/postgresql@autosnap_2022-12-24_00:00:00_daily 2.54G - 125G - data/sync/dali/postgresql@autosnap_2022-12-25_00:00:02_daily 2.51G - 125G - data/sync/dali/postgresql@autosnap_2022-12-26_00:00:01_daily 0B - 125G - data/sync/dali/postgresql/wal 447M 294G 68.5M none data/sync/dali/postgresql/wal@autosnap_2022-12-20_00:00:00_daily 49.0M - 49.1M - data/sync/dali/postgresql/wal@autosnap_2022-12-21_00:00:00_daily 66.9M - 67.0M - data/sync/dali/postgresql/wal@autosnap_2022-12-22_00:00:01_daily 76.2M - 76.2M - data/sync/dali/postgresql/wal@autosnap_2022-12-23_00:00:01_daily 65.8M - 65.8M - data/sync/dali/postgresql/wal@autosnap_2022-12-24_00:00:00_daily 60.2M - 76.2M - data/sync/dali/postgresql/wal@autosnap_2022-12-25_00:00:02_daily 60.9M - 77.0M - data/sync/dali/postgresql/wal@autosnap_2022-12-26_00:00:01_daily 0B - 68.5M -mentioned in commit swh-sysadmin-provisioning@0350a4d5
removed state:wip label
mentioned in issue #4816 (closed)
