Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • S sysadm-environment
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 199
    • Issues 199
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Model experiments
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Platform
  • Infrastructure
  • sysadm-environment
  • Issues
  • #3833

Dedicate one admin host to centralize administration dbs

Centralize the postgresql databases used for admin tools into a single dedicated host (on which we would be able to do more proper backups and monitoring).

This is also the occasion to use the latest postgresql versions (each have their own and inconsistent versions).

Impacted services:

  • hedgedoc (host: bardo, db: postgres-12)
  • netbox (host: bojimans, db: postgres-11)
  • grafana (host: pergamon, db: postgres-11)
  • sentry (host: riverside, db: postgres-12)
  • keycloak (host: kelvingrove, db: postgres-12)

This is the size of the current databases:

Database Size
hedgedoc 42MB
netbox 22MB
grafana 18MB
sentry 99GB
keycloak 15MB

The sentry database will force us to reserve a large amount of disk for the new server

Plan:

  • Leave services' configuration untouched to use local db

  • infra/puppet/puppet-swh-site!467, swh-sysadmin-provisioning!57 (closed): First create host database machine (vm).

  • #3833 (closed): Create zfs data mount point (for the dbs' data)

  • infra/puppet/puppet-swh-site!471: Declare a dedicated puppet profile, this lists all required dbs to create (using profile::postgresql::server).

  • swh-sysadmin-provisioning!57 (closed): terraform (/vagrant) to boostrap (this applies puppet so the dbs get created)

  • #3833 (closed): firewall: Open flux from vlan 440 (bojimans, kelvingrove, riverside, #pergamon) to vlan 442, port 5432

  • for each service in {infra/puppet/puppet-swh-site!468: netbox, infra/puppet/puppet-swh-site!469: hedgedoc, #3817 (closed): grafana, infra/puppet/puppet-swh-site!470: sentry, rSPSITE2b8a33e79d6e49554339e3b70134eb84e8cad7cf: keycloak}:

    • Stop the service (we don't have incremental dump so stop the service first)

    • Export and mount back data dump from old db to the new one

    • Adapt configuration to switch to the new db

    • puppet apply to restart service (which now uses the new db)

    • Ensure service is still ok

Annex actions (outside the scope of this task, like #3817 (closed)):

  • #3850: Move services {netbox, sentry, keycloak} in the admin vlan (442) and behind the reverse proxy
  • #3849 (closed): Clean up leftovers after migration

Related to infra/puppet/puppet-swh-site!463


Migrated from T3833 (view on Phabricator)

Edited Oct 18, 2022 by Antoine R. Dumont
Assignee
Assign to
Time tracking