Deploy swh.core > v2.4

Plan:

• D7641: Unstuck master build
• Tag v2.5
• Unstuck debian build if needed

staging:

• Stop workers [1]
• Upgrade storage node (storage1.staging)'s swh.core
• Upgrade storage node's swh.storage v1.3.2 while at it
• Restart gunicorn-swh-storage, gunicorn-swh-objstorage
• #4189 (closed): Upgrade loader dependencies, reboot workers and activate back puppet [2]
• Upgrade indexer node (storage1.staging)'s swh.core
• Restart gunicorn-swh-indexer-storage
• Upgrade scheduler node (scheduler0.staging)'s swh.core
• Restart gunicorn-swh-scheduler
• Upgrade archive node (webapp.staging)'s swh.core
• Restart gunicorn-swh-webapp gunicorn-swh-storage
• Upgrade deposit node (deposit.staging)'s swh.core
• Restart gunicorn-swh-deposit
• Upgrade vault node (vault.staging)'s swh.core
• Restart gunicorn-swh-vault
• Upgrade counters node (counters0.staging)'s swh.core
• Restart gunicorn-swh-counters

prod:

• Stop workers [1]

• Upgrade storage node (saam)'s swh.core

• Upgrade storage node's swh.storage v1.3.2 while at it

• Restart gunicorn-swh-storage, gunicorn-swh-objstorage

• #4189 (closed): Upgrade loader dependencies, reboot workers and activate back puppet

• Upgrade indexer node (saam)'s swh.core

• Restart gunicorn-swh-indexer-storage

• Upgrade scheduler node (saatchi)'s swh.core

• Restart gunicorn-swh-scheduler

• Upgrade archive node (moma, webapp1)'s swh.core

• Restart gunicorn-swh-webapp gunicorn-swh-search gunicorn-swh-storage gunicorn-swh-indexer-storage gunicorn-swh-objstorage

• Upgrade deposit node (moma)'s swh.core

• Restart gunicorn-swh-deposit

• Upgrade vault node (vangogh)'s swh.core

• Restart gunicorn-swh-vault

• Upgrade counters node (counters1)'s swh.core

• Restart gunicorn-swh-counters

• [1]

# staging:
WORKERS=-w @staging-workers
# prod
# WORKERS=-w @swh-workers -w @swh-workers-large
clush -b $WORKERS 'puppet agent --disable "Deploy new swh.core version"; systemctl stop cron; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl disable $unit; done; systemctl stop "swh-worker@*"'

• [2]

# since most services are stopped
$ clush -b $WORKERS \
  "DEBIAN_FRONTEND=noninteractive apt-get update && apt-get upgrade -y && reboot"
# once all booted
$ clush -b $WORKERS 'uptime'
# activate back everything
$ clush -b $WORKERS \
  'puppet agent --enable; puppet agent --test; systemctl default'

---

Migrated from T4197 (view on Phabricator)