Instabilities after the deployment of swh v309

After the latest deployment various issues happened in the production environment:

• Postgresql connection pool full
• Problem to interact with the azure object storage
• #5337 (closed): read-only instance fails to start on writable check failing

added activity::Deployment incident labels

assigned to @vsellier

marked this incident as related to #5329 (closed)

added an incident timeline event

edited the event time/date on incident timeline event

added an incident timeline event

added an incident timeline event

edited the event time/date on incident timeline event

added an incident timeline event

edited the text on incident timeline event

edited the text on incident timeline event

edited the text on incident timeline event

added an incident timeline event

added an incident timeline event

added an incident timeline event

edited the event time/date on incident timeline event

added an incident timeline event

The fix of the azure access issue: swh/devel/swh-objstorage!194 (merged)

Looks like a regression of swh/devel/swh-objstorage@cd672034

Unfortunately, the exception was not recorded in ELK due to an opentelemety parsing issue #5335

edited the text on incident timeline event

Regarding the connection number issue, there is no significant connection peak on postgresql in the interval

• ~ 175 connections to 5433
• ~ 135 connections to 5434
• ~ 20 connections to 5435

It doesn't reflect the pgbouncer activity but there is no other stats available.

To have some metrics for pgbouncer, the dedicated prometheus exporter should be installed on albertina and massmoca (and staging): #5336

edited the text on incident timeline event

Sentry Issue: SWH-LOADER-SVN-J2

Sentry Issue: SWH-LOADER-SVN-J3

Sentry Issue: SWH-STORAGE-2X7V

Sentry Issue: SWH-SCHEDULER-CX

Sentry Issue: SWH-WEBAPP-5X5

changed the description

changed title from Instabilities after the deployment of swh v310 to Instabilities after the deployment of swh v309

mentioned in issue #5329 (closed)

mentioned in commit swh/devel/swh-objstorage@12bc3ba8

mentioned in merge request swh/devel/swh-objstorage!195 (closed)

added environment: production environment: staging labels

changed the description

Connection limit on pgbouncer: swh/infra/puppet/puppet-swh-site@fa18c480

swh-objstorage check_config issues: swh/devel/swh-objstorage!194 (merged) released in swh.objstorage 3.1.2 and deployed

closed

changed the incident status to Resolved by closing the incident