Make opam shared root initialization more robust

The way we initialize the opam shared root on workers, through multiple timer units, is pretty brittle. A lot of the services actually fail to run because of a race condition or because of "lack of a switch", whatever that means.

Jan 03 00:00:00 worker01 systemd[1]: Started Software Heritage Manage OPAM shared state (coq.inria.fr).
Jan 03 00:00:01 worker01 opam-manage-shared-state.sh[1475377]: [WARNING] No switch is currently set, perhaps you meant '--set-default'?
Jan 03 00:00:02 worker01 opam-manage-shared-state.sh[1475377]: [coq.inria.fr] Initialised
Jan 03 00:00:02 worker01 opam-manage-shared-state.sh[1475377]: [ERROR] No switch is currently set. Please use 'opam switch' to set or install a switch
Jan 03 00:00:02 worker01 systemd[1]: opam-manage-shared-state-coq.inria.fr.service: Main process exited, code=exited, status=50/n/a
Jan 03 00:00:02 worker01 systemd[1]: opam-manage-shared-state-coq.inria.fr.service: Failed with result 'exit-code'.

Instead of having separate timer units, we should probably have a single one which would run a script that would create the default root first, then run snippets updating the created root for each separate instance afterwards.

• use set -e in the main script if it's not already set
• make the update script update all instances at once, instead of a single one separately: write a main script which handles the creation of the opam root and the update of the main instance, and run snippets generated for each other instance, using run-parts
• make sure the worker only starts after a successful run of the main service
• make sure the timer unit runs at different times, rather than all at midnight
• consider moving the shared root to /var/tmp to avoid it being blown away by reboots (probably not needed if we make sure the service dependencies are correct)

---

Migrated from T3826 (view on Phabricator)

added Opam System administration priority:High labels

marked the checklist item use set -e in the main script if it's not already set as completed

marked the checklist item make the update script update all instances at once, instead of a single one separately: write a main script which handles the creation of the opam root and the update of the main instance, and run snippets generated for each other instance, using run-parts as completed

marked the checklist item make sure the worker only starts after a successful run of the main service as completed

marked the checklist item make sure the timer unit runs at different times, rather than all at midnight as completed

marked the checklist item consider moving the shared root to /var/tmp to avoid it being blown away by reboots (probably not needed if we make sure the service dependencies are correct) as completed

Deployed. And alerts are no longer now [1]

consider moving the shared root to /var/tmp to avoid it being blown away by reboots (probably not needed if we make sure the service dependencies are correct)

I did not go the path change way since that impacts current code in both lister and loader (which are running with the default fallback values). And the dependencies in the systemd are correct as per our iteration on the diff.

• [1]

09:25 <+swhbot> icinga RECOVERY: service check_systemd on worker0.internal.staging.swh.network is OK: SYSTEMD OK - all
09:25 <+swhbot> icinga RECOVERY: service check_systemd on worker2.internal.staging.swh.network is OK: SYSTEMD OK - all
09:25 <+swhbot> icinga RECOVERY: service check_systemd on worker1.internal.staging.swh.network is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker07.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker10.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker16.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker01.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker13.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker12.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker08.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker06.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker09.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker04.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker14.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker02.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker11.softwareheritage.org is OK: SYSTEMD OK - all
09:26 <+swhbot> icinga RECOVERY: service check_systemd on worker03.softwareheritage.org is OK: SYSTEMD OK - all
09:27 <+ardumont> good workers!
09:27 <+swhbot> icinga RECOVERY: service check_systemd on worker05.softwareheritage.org is OK: SYSTEMD OK - all
09:27 <+swhbot> icinga RECOVERY: service check_systemd on worker15.softwareheritage.org is OK: SYSTEMD OK - all

added state:wip label

removed state:wip label

closed