Objstorage volume size in staging environment is critical

On storage1.internal.staging.swh.network:

root@storage1:~# df -hT | awk 'NR==1 || /zfs/' | grep objects
Filesystem                Type      Size  Used Avail Use% Mounted on
data/objects              zfs        25T   23T  2.3T  91% /srv/softwareheritage/objects

mentioned in issue #5260

changed the description

status:

root@storage1:~# date; echo; df -hT | head -1; df -hT | awk 'NR==1 || /zfs/' | grep objects
Fri 01 Mar 2024 09:08:42 AM UTC

Filesystem                Type      Size  Used Avail Use% Mounted on
data/objects              zfs        28T   27T  989G  97% /srv/softwareheritage/objects

We've got still running quite a few loaders. I'll stop some of them and reduce others. To give us a bit of breath to work on #5256 (closed).

done (grafana tagged too)

mentioned in commit swh/infra/ci-cd/swh-charts@8b9001c2

mentioned in commit swh/infra/ci-cd/swh-charts@9917758a

Dashboard to follow on the ingestion status for staging [1]

[1] https://grafana.softwareheritage.org/goto/XjjwpeoIk?orgId=1

Current work to stop writing to that objstorage is tracked [1]

That should be able to wait for next week at current speed (~7Gi/h) [2].

[2]

root@storage1:~# date; echo; df -hT | head -1; df -hT | awk 'NR==1 || /zfs/' | grep objects
Fri 01 Mar 2024 04:19:10 PM UTC

Filesystem                Type      Size  Used Avail Use% Mounted on
data/objects              zfs        28T   27T  961G  97% /srv/softwareheritage/objects

[1] #5260

It seems stopping some loaders did the trick to slow down the ingestion in staging ;)

root@storage1:~# date; echo; df -hT | head -1; df -hT | awk 'NR==1 || /zfs/' | grep objects
Mon 04 Mar 2024 08:48:23 AM UTC

Filesystem                Type      Size  Used Avail Use% Mounted on
data/objects              zfs        28T   27T  884G  97% /srv/softwareheritage/objects

root@storage1:~# date; echo; df -hT | head -1; df -hT | awk 'NR==1 || /zfs/' | grep objects
Tue 05 Mar 2024 11:19:20 AM UTC

Filesystem                Type      Size  Used Avail Use% Mounted on
data/objects              zfs        28T   27T  856G  97% /srv/softwareheritage/objects

The storage1 objstorage1 is no longer writable now. A new r/w objstorage has been deployed on db1.

closed