Large number of swh-storage timeouts since 2023-11-24

Sentry shows a large number of database timeouts started happening yesterday evening, around 21:00-22:00 (unclear what timezone, probably Paris):

(ignore the TypeErrors, it's a different issue caused by timeouts: swh/devel/swh-storage!1097 (merged) )

created branch 5167-large-number-of-swh-storage-timeouts-since-2023-11-24 to address this issue

changed the description

assigned to @vsellier

There was indeed a higher rate of error during between the 2023-11-24 and and the 2023-11-27

It the result of a high load on the database server

The trigger is the backup (+ the usual high load of course)

pgbackrest@banco:~$ pgbackrest info
stanza: main
    status: ok
    cipher: none

    db (current)
        wal archive min/max (12): 0000000100033B67000000ED/0000000100033CD800000089

        full backup: 20231124-204202F
            timestamp start/stop: 2023-11-24 20:42:02+00 / 2023-11-27 03:43:54+00
            wal start/stop: 0000000100033B67000000ED / 0000000100033C9C00000000
            database size: 35927.3GB, database backup size: 35927.3GB
            repo2: backup size: 13019.5GB

@olasd Do you think it should be possible to reduce the pressure of the backup on the database? The only leverage I see is by reducing the number of process-max?

Before reducing the load (and therefore, probably, substantially increasing the duration of backups), I'd much rather we tried to increase the timeouts on the specific functions impacted by a bit.

Afaict the backups are throttled by the 1Gbps link to the backup server, and keeping the duration of backups short seems desirable overall.

(more precisely, I'm not sure that the timeouts are caused by the load of the backup itself, or by the fact that the database is in concurrent backup mode, with increased WAL traffic, for the duration of the backup)

To test this out, I've reduced the cron entry to 16 processes instead of 32 for tonight's run.

Thanks, as you said, with the reduction of the parallelism, the backup seems to be more gentle with the database.

it's one day longer but there is no more timeout increase.

closed