Migrate production database servers to bullseye

This focuses on migrating the release of the (db) nodes from buster to bullseye. This does not migrate the postgresql version in itself. This is another dedicated task [1].

• [1] swh/infra/sysadm-environment#2581

Servers to migrate:

• db1.internal.staging.swh.network
• belvedere.internal.softwareheritage.org
• somerset.internal.softwareheritage.org

Plan: staging: Moved to this subtask #3813 (closed)

Production:

• upgrade somerset

  • switch the webapp to belvedere
  • on somerset
  • disable puppet
  • stop and disable postgresql
  • perform the last buster upgrade
  • reboot (restarted recently)
  • perform the bullseye upgrade
  • reboot
  • restart and enable postgresql
  • check the replication with belvedere is ok
  • switch back the webapp to somerset

• upgrade of belvedere

  • add a notification in the status.io page

A database upgrade is scheduled the XXXX-XX-XX between XX:XX and XX:XX 
Some service disruptions can occur during this period

Impacted services:
- archive.softwareheritage.org
- Save code now
- Source code crawler
- deposit

- [x] connect to the idrac: https://swh9-adm.inria.fr/
- [x] stop the loaders and listers workers
- [x] stop the indexers
- [x] stop the scheduler runners + those in the tmux in saatchi
- [x] **ensure the provenance experiment is stopped**
- [x] on belvedere:
    - [X] stop puppet
    - [X] ~~stop and disable postgresql**s ** (to avoid the restarts after the server reboots)~~ can be ignored
    - [x] ~~perform the last upgrade of buster~~
    - [x] ~~reboot~~
    - [X] upgrade to bullseye
    - [X] reboot        
    - [X] check everything is going well after the reboot
    - [X]  ~~start and enable the postgresql servers~~
    - [X] check the replication to somerset is ok
    - [X] reactivate puppet
- [x] restart stopped services

---

Migrated from T3801 (view on Phabricator)