Dedicate one admin host to centralize administration dbs
Centralize the postgresql databases used for admin tools into a single dedicated host (on which we would be able to do more proper backups and monitoring).
This is also the occasion to use the latest postgresql versions (each have their own and inconsistent versions).
Impacted services:
- hedgedoc (host: bardo, db: postgres-12)
- netbox (host: bojimans, db: postgres-11)
- grafana (host: pergamon, db: postgres-11)
- sentry (host: riverside, db: postgres-12)
- keycloak (host: kelvingrove, db: postgres-12)
This is the size of the current databases:
| Database | Size |
|---|---|
| hedgedoc | 42MB |
| netbox | 22MB |
| grafana | 18MB |
| sentry | 99GB |
| keycloak | 15MB |
The sentry database will force us to reserve a large amount of disk for the new server
Plan:
-
Leave services' configuration untouched to use local db -
infra/puppet/puppet-swh-site!467, swh-sysadmin-provisioning!57 (closed): First create host database machine (vm). -
#3833 (closed): Create zfs data mount point (for the dbs' data) -
infra/puppet/puppet-swh-site!471: Declare a dedicated puppet profile, this lists all required dbs to create (using profile::postgresql::server). -
swh-sysadmin-provisioning!57 (closed): terraform (/vagrant) to boostrap (this applies puppet so the dbs get created) -
#3833 (closed): firewall: Open flux from vlan 440 (bojimans, kelvingrove, riverside, #pergamon) to vlan 442, port 5432 -
for each service in {infra/puppet/puppet-swh-site!468: netbox, infra/puppet/puppet-swh-site!469: hedgedoc, #3817 (closed): grafana, infra/puppet/puppet-swh-site!470: sentry, rSPSITE2b8a33e79d6e49554339e3b70134eb84e8cad7cf: keycloak}:
-
Stop the service (we don't have incremental dump so stop the service first)
-
Export and mount back data dump from old db to the new one
-
Adapt configuration to switch to the new db
-
puppet apply to restart service (which now uses the new db)
-
Ensure service is still ok
-
Annex actions (outside the scope of this task, like #3817 (closed)):
- #3850: Move services {netbox, sentry, keycloak} in the admin vlan (442) and behind the reverse proxy
- #3849 (closed): Clean up leftovers after migration
Related to infra/puppet/puppet-swh-site!463
Migrated from T3833 (view on Phabricator)