Install grafana on its own server

Install grafana on its own server.

• in the admin vlan

• behind rp1

• bullseye

• Decide what to do with the swh-grafanalib-dashboards package to avoid manual installations (https://forge.softwareheritage.org/source/swh-grafana-dashboards/):

  • upgrade to bullseye ?
  • migrate to puppet ?

---

Migrated from T3817 (view on Phabricator)

marked this issue as related to swh/meta#3808

added System administration priority:Low state:wip labels

mentioned in commit swh-sysadmin-provisioning@6230309a

The diffs are ready to be reviewed. The migration will be performed at the beginning of January.

Actions to perform:

• land the infra/puppet/puppet-swh-site!463 diff when validated
• provision the new vm with terraform
• apply puppet on rp1
• Create the following rules on the firewall
  • VLAN442:
    • icinga server -> grafana server:postgresql (to validate if it's really needed)
  • VLAN440:
    • grafana server -> prometheus server:prometheus port
    • grafana server -> elasticsearch nodes:elasticsearch port
• dump the grafana database on pergamon

sudo -i -u postgres pg_dump -c -C grafana > grafana.sql

(-c -C to recreate the database on the new vm as it will be already created by puppet)

• import the database on grafana0

cat grafana.sql | sudo -i -u postgres pgsql

• check evrything is ok by overriding adding the following line in your /etc/hosts

192.168.50.20 grafana.softwareheritage.org`

• if everything if ok, change the public dns entry on the softwareheritage.org domain to

grafana CNAME 1800 swh-rproxy3.inria.fr.

If everything is ok, prepare a diff to cleanup the prometheus node / manual cleanup

• remove the grafana vhosts from apache
• remove the following packages
  • grafana
  • swh-grafanalib-dashboards
• drop the grafana database from postgresql
• remove the following directories:
  • /var/lib/grafana
  • /etc/grafana

We will wait for the availability of the dedicated admin database server (#3833 (closed)) before moving grafana. It will avoid to move the data 2 times and will upgrade the database to the last postgresql version.

added priority:Normal label and removed priority:Low label

• grafana0 server created
• configuration applied on the reverse proxy
• network configuration checks:
  • grafana0 -> pergamon:9090 (prometheus): Allowed for all in a floating rule
  • grafana0 -> esnode[1-3]:9200 (elasticsearch queries):
• public DNS updated to use rp1 (swh-rproxy3.inria.fr) as the public entry point
• pergamon apache configuration temporary updated to point to the new grafana0 backend server ( and add a temporary rule on the firewall to allow it)

• pergamon cleanup done with infra/puppet/puppet-swh-site!464.

Apache vhosts were automatically removed

The database removal will be handled in #3849 (closed)

removed state:wip label

closed

mentioned in issue #3833 (closed)

mentioned in issue #3850