Deploy swh-loader-svn v1.6.0 to production

added SVN Loader System administration activity::Deployment priority:Normal labels

changed the description

Problem during the docker image build [1]:

14:30:06  Head "https://container-registry.softwareheritage.org/v2/infra/swh-apps/rsvndump-base/manifests/latest": denied: access forbidden

[1] https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/39/console

mentioned in commit swh-apps@ca564d01

mentioned in commit swh-apps@d9c8d20c

Pushed rsvndump in gitlab registry and adapted both image Dockerfiles to use the correct rsvndump gitlab registry image.

Build of the swh-loader-svn app image is ok now.

[1] https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/40/console

mentioned in commit ardumont/swh-charts@c544eb91

marked the checklist item build images (swh-loader-svn, swh-loader-highpriority) as completed

marked the checklist item swh-charts: Update versions as completed

marked the checklist item (ongoing) Let argocd deploy new versions as completed

marked the checklist item Checks as completed

changed the description

Upgrade [1]
Restart services [2]

[1]

root@pergamon:~# clush -b -w @swh-workers 'env DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -y upgrade'
root@pergamon:~# clush -b -w @swh-workers 'dpkg -l python3-swh.loader.svn | grep ii'
---------------
worker[01-16] (16)
---------------
ii  python3-swh.loader.svn 1.6.0-1~swh1~bpo10+1 all          Software Heritage Loader Svn

[2] (dedicated systemd services)

root@pergamon:~# clush -b -w @swh-workers "systemctl restart swh-worker@loader_svn"
root@pergamon:~# clush -b -w @swh-workers "systemctl restart swh-worker@loader_high_priority"  # uses svn loader (but not only, loader_git, loader_hg, ...)

marked the checklist item workers: Upgrade python3-swh.loader.svn as completed

marked the checklist item Restart services swh_worker@loader_{svn,high_priority} as completed

changed milestone to %MRO 2023

marked the checklist item Checks as completed

Checking services' status renders origins being ingested.

[1]

root@pergamon:~# clush -b -w @swh-workers "systemctl status swh-worker@loader_svn"
root@pergamon:~# clush -b -w @swh-workers "systemctl status swh-worker@loader_high_priority"

closed

I'm not sure what happened when you restarted the production services, but they all got restarted by the crontab rather than by (or in addition to ?) your systemctl command. (noticed as a flood of mails to root by @guillaume)

Did you stop the services manually before the upgrade?

Did you stop the services manually before the upgrade?

No, i did not.

I've "restart"ed the systemd services after the package upgrade. It stuck because some loading were still happening. And it finally restarted after the 15min timeout.

I've "restart"ed the systemd services after the package upgrade. It stuck because some loading were still happening. And it finally restarted after the 15min timeout.

OK, then I think what's happening is that, while the service is stopping:

systemctl is-enabled foo.service returns true
systemctl is-active foo.service returns false (so the cron triggers a systemctl start foo.service, and complains)

But practically the start command is deferred until after the service times out during stop (and gets coalesced with the start operation that has been triggered by the admin-provided restart command)

That makes sense.

Now, given that, I still don't know how to avoid the spammy situation. It does the cron/email thing each time I upgrade a package with associated long-running task/service.

It's a bug in the check-restart script, it should avoid doing anything if there's a restart job queued.

To avoid spamming on upgrades I've now grown accustomed to using the following (pretty silly, I'll admit) procedure

before the upgrade:
- puppet agent --disable "upgrading $service" (this prevents the services from being re-enabled behind my back)
- systemctl disable the relevant services, without stopping them (the only side effect of this operation is disabling the ping-restart cron)
- use the celery remote control to stop consuming the queues
- wait a sensible amount of time (10-15 minutes?)
do the upgrade with apt
after the upgrade
- systemctl stop --no-block the services (this returns immediately)
- wait a bit for the services that were already done to stop
- systemctl kill --signal=9 --kill-who=all the (remaining) services
- run puppet agent --enable && puppet agent -t; systemctl default (this re-enables the services, which re-enables the cron, and systemctl default restarts them)

The celery remote control is an ungodly command that only really lives in my zsh history (it uses some zsh-specific join syntax)


# adapt this for the right hosts
hosts=(worker{01..16}.internal.softwareheritage.org)
# and the right worker instances
workers=(checker_deposit lister loader_{archive,bzr,cran,cvs,debian,deposit,git,mercurial,nixguix,npm,oneshot,opam,pypi,svn})

# these are generic
all_instances=(${^workers}@${^hosts})

for queue in `SWH_CONFIG_FILENAME=~/work/swh-environment/scratch/scheduler.yml celery -A swh.scheduler.celery_backend.config.app inspect -d  ${(j:,:)all_instances} active_queues -j | head -n +1 | jq -r '[. | to_entries | .[].value | flatten | .[].name] | unique | .[]' | grep -v oneshot3`; do
    SWH_CONFIG_FILENAME=~/work/swh-environment/scratch/scheduler.yml celery -A swh.scheduler.celery_backend.config.app control -d  ${(j:,:)all_instances} cancel_consumer $queue
done

scheduler.yml contains a celery: task_broker: amqp:/// entry with credentials to write remote control commands to the (production or staging) rabbitmq broker.

Clearly not really tractable for what should be a simple deployment task...

Thanks for taking the time to describe all this ;p

Deploy swh-loader-svn v1.6.0 to production

Designs

Child items ...

Activity