Configure db1 as new staging db server

added System administration priority:Normal state:wip labels

Create the postgresql:5434 dataset

zfs create data/postgres-secondary-12 -o mountpoint=/srv/softwareheritage/postgres/12/secondary

Create the postgresql:5435 dataset

zfs create data/postgres-indexer-12 -o mountpoint=/srv/softwareheritage/postgres/12/indexer

check datasets :

root@db1:~# zfs list -t all
NAME                         USED  AVAIL     REFER  MOUNTPOINT
data                        1.20M  26.4T       96K  /data
data/postgres-indexer-12      96K  26.4T       96K  /srv/softwareheritage/postgres/12/indexer
data/postgres-main-12         96K  26.4T       96K  /srv/softwareheritage/postgres/12/main
data/postgres-misc           104K  26.4T      104K  /srv/softwareheritage/postgres
data/postgres-secondary-12    96K  26.4T       96K  /srv/softwareheritage/postgres/12/secondary

For the puppet part, the actual staging configuration needs some adaptations as the configuration install postgresql on version 11 and 13. Another point is the different clusters are not managed by puppet but it's the same for the production.

At least, the configuration needs to be adapted to install only the version 12 to have the same version as in the production environment (except for the indexer still using the version 11). We will dig with @ardumont if it's also possible to manage the 3 psql clusters via puppet in a reasonable amount of time

For the puppet part, the actual staging configuration needs some adaptations as the configuration install postgresql on version 11 and 13

Fixed by infra/puppet/puppet-swh-site!240.

The puppetlabs-postgresql module doesn't allow to manage several postgresql clusters. We have made the tradeoff to use only one cluster on db1 at the beginning to be able to deploy db1 via puppet as it's the priority. The module will be extended or replaced by something else later.

mentioned in commit swh/devel/swh-core@74544839

mentioned in commit swh/devel/swh-environment@2a5d9a27

mentioned in commit swh/devel/swh-indexer@d910ad3e

Road so far, after the puppet run which creates the dbs with guest, postgres and swh-admin users.

As tryouts, from staging-db1 node (and a bit on the actual db1, which we'll finish during the day):

$ cat init-db.sh
#!/usr/bin/env bash

set -x

export PGUSER=postgres
# export PGHOST=$(ip a | grep eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1)
export PGHOST=db1.internal.staging.swh.network
export PGPORT=5432
# export PGPASSWORD=swh::deploy::db::postgres::password
export PGPASSWORD=<insert-postgres-pass-here>

MODULE=${1}

if [ "${MODULE}" == "storage" ]; then
   DBNAME="swh"
else
   DBNAME="swh-${MODULE}"
fi

# Connect as super-admin to init the extensions
swh db init-admin --db-name $DBNAME $MODULE   # should be changed to init-admin

# Connect as owner of the db to init the schema
export PGUSER=$DBNAME
export PGPASSWORD=${2}
swh db init --db-name $DBNAME $MODULE

# Connect as owner to grant read access to schema to guest user
psql -c "grant select on all tables in schema public to guest;"

The actual initialization calls to bootstrap the different schema below (the second parameter is the password, for vagrant node, coming from the censored private data repository, for the actual production, this will be adapted accordingly):

$ ./init-db.sh scheduler swh::deploy::scheduler::db::password
DONE database for scheduler initialized at version 17
$ ./init-db.sh indexer swh::deploy::indexer::storage::db::password
DONE database for indexer initialized at version 132
$ ./init-db.sh vault swh::deploy::vault::db::password
DONE database for vault initialized at version 1
$ ./init-db.sh storage swh::deploy::storage::db::password
DONE database for storage initialized (flavor default) at version 163

Note:

swh db create is used for now, the code on the node is patched to comment out the sql create database call to prevent breaking (D4374).
the swh-admin superuser is used to allow privilege installation setup (infra/puppet/puppet-swh-site!243)

Other dbs have their own initialization way:

deposit: not the same initialization (no sql files, django orm inside) https://docs.softwareheritage.org/devel/swh-deposit/sys-info.html#environment-production
lister: not the same initialization (no sql files, sqlalchemy...) https://intranet.softwareheritage.org/wiki/Deploying_a_lister#Update_the_listers_database_model

mentioned in commit swh/devel/swh-vault@7b4b59c6

Staging nodes are now using storage1 (and db1). So db0 and storage0 can be decomissionned (and thus orsay \o/).

We triggered some loading (swh loader run git..., some are still running btw, linux, nixpkgs, guix, ...). Icinga checks have been fixed a bit... (firewall rule update, ...)

We need to shake some more other services (save code now, cooking task, trigger listings, ... )

Following the diff infra/puppet/puppet-swh-site!244, the zfs dsatasets were reconfigured tobe mounted on the /srv/softwareheritage/postgres/* :

systemctl stop postgresql@12-main
zfs set mountpoint=none data/postgres-indexer-12
zfs set mountpoint=none data/postgres-secondary-12
zfs set mountpoint=none data/postgres-main-12
zfs set mountpoint=none data/postgres-misc

zfs set mountpoint=/srv/softwareheritage/postgres data/postgres-misc
zfs set mountpoint=/srv/softwareheritage/postgres/12/indexer data/postgres-indexer-12
zfs set mountpoint=/srv/softwareheritage/postgres/12/secondary data/postgres-secondary-12
zfs set mountpoint=/srv/softwareheritage/postgres/12/main data/postgres-main-12

systemctl start postgresql@12-main

It must be done in this order because there is a hierarchical mount between postgres/ and postgres/*

closed

Configure db1 as new staging db server

Designs

Child items ...

Activity