Installation of the new provenance server

• [in progress] Provision the server in the inventory (rack position, idrac ips, ips, ...)
• [Done] notify the dsi of the delivery + give the information for the installation
• define the role in puppet
• Install the server

List of packages amended along the way:

• python3-virtualenvwrapper
• libpq-dev
• arcanist

Some more configuration:

• install scripts ./create-db.sh and ./drop-db.sh to ease db maintenance for @aeviso to drop/create dbs

ardumont@met:~% cat drop-db.sh
#!/usr/bin/env bash

DBNAME=$1

sudo -i -u postgres dropdb $DBNAME
ardumont@met:~% cat create-db.sh
#!/usr/bin/env bash

DBNAME=$1

sudo -i -u postgres createdb -p 5433 --lc-ctype=C.UTF-8 -T template0 -O swh-provenance $DBNAME

• Provision 10 dbs through puppet (so pgbouncer is configured as well) (infra/puppet/puppet-swh-site!420)
• at some point, dedicate some vms to andres so he can experiment from those, passing through the internal network (without the vpn)

---

Migrated from T3487 (view on Phabricator)

added System administration priority:Normal state:wip labels

changed the description

@jayeshv @aeviso @douardda @olasd have you an idea of what should be installed on the server and who will operate what will be on it?

It's not completely clear for me if this server will be a sandbox/staging or a production server.

@vsellier I am not sure about this. The idea is to use this machine as the production server. (I guess this will host either postgres or mongodb after we decide on a preferred backend. But that is going to take some time) @olasd or @aeviso will know better.

changed the description

yes the idea is to have a beefy enough machine to perform full-size experiments on, that can then be (part of) the production infrastructure dedicated to the provenance index.

As see with @aeviso , we will install the following components on the server (the os will be debian11)

• rabbitmq
• postgresql:13
  • a default swh-storage database will be managed by puppet
  • 1000 parallel connections allowed
  • shared_buffers 50go
• docker

*- WIP -* Additional standard packages:

• zfs, the datasets will be configured by sysadms
• default statsd/prometheus exporter plugged on the main prometheus

! In #3487 (closed), @vsellier wrote:

• postgresql:13
  • 1000 parallel connections allowed

wouldn't it better to use pgbouncer (or similar)?

yes pgbouncer will be used and it's configured by default to 2000 // connections I don't know the kind of load the provenance client will generate but the default 100 connections allowed by postgres will be probably too short and needed to be increased too

The server is installed. It remains few task to perform manually:

• configure the zfs datasets (will configure 2 mirror pool for ~12To available, tell me if it's not what it's expected)
• build few missing packages for bullseye (relative to the monitoring: prometheus-rabbitmq-exporter, prometheus-statsd-exporter, journalbeat)
• configure a rabbitmq admin user

@aeviso you should be able to connect on the server.

The hostname is met.internal.softwareheritage.org

I forgot to mention there is a gift from dell on the server: an additional 600Go 10rpm disk

changed the description

The zfs pool and dataset are configured:

• pool configuration

## nvme drives pool
#zpool create data mirror nvme-eui.36315030525005540025384500000003 nvme-eui.36315030525005800025384500000003 mirror nvme-eui.36315030525005620025384500000003 nvme-eui.36315030525005890025384500000003

## bonus pool
# zpool create data-hdd wwn-0x5000c500dea6c533

• postgresql dataset //move the current postgresql content away and copy it on the new directory after //

# zfs create -o mountpoint=/srv/softwareheritage/postgres/13/main -o atime=off -o relatime=on data/postgresql

• status

root@met:~# zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data      11.6T  52.0M  11.6T        -         -     0%     0%  1.00x    ONLINE  -
data-hdd   556G   114K   556G        -         -     0%     0%  1.00x    ONLINE  -
root@met:~# zpool list -v
NAME                                            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data                                           11.6T  52.4M  11.6T        -         -     0%     0%  1.00x    ONLINE  -
  mirror                                       5.81T  26.5M  5.81T        -         -     0%  0.00%      -  ONLINE  
    nvme-eui.36315030525005540025384500000003      -      -      -        -         -      -      -      -  ONLINE  
    nvme-eui.36315030525005800025384500000003      -      -      -        -         -      -      -      -  ONLINE  
  mirror                                       5.81T  25.9M  5.81T        -         -     0%  0.00%      -  ONLINE  
    nvme-eui.36315030525005620025384500000003      -      -      -        -         -      -      -      -  ONLINE  
    nvme-eui.36315030525005890025384500000003      -      -      -        -         -      -      -      -  ONLINE  
data-hdd                                        556G   114K   556G        -         -     0%     0%  1.00x    ONLINE  -
  wwn-0x5000c500dea6c533                        556G   114K   556G        -         -     0%  0.00%      -  ONLINE  

root@met:~# zfs list
NAME              USED  AVAIL     REFER  MOUNTPOINT
data             51.8M  11.3T       24K  /data
data-hdd          114K   539G       24K  /data-hdd
data/postgresql  51.6M  11.3T     51.6M  /srv/softwareheritage/postgres/13/main

changed the description