db1.internal.admin.swh.network (dali) gets connection-starved

We're regularly getting emails from netbox that dali's postgresql server gets connection-starved.

We should check what uses that many connections and assess whether we should increase max_connections or fix the offending user.

Nothing seems particularly off, I think we're just hitting the default 100 connection limit (and the prometheus-sql-exporter is a hog):

postgres=# select datname, application_name, count(*) from pg_stat_activity group by 1,2;
 datname  |    application_name     | count 
----------+-------------------------+-------
 grafana  |                         |     2
 sentry   |                         |    33
 hedgedoc | prometheus-sql-exporter |     5
 postgres | psql                    |     1
 netbox   | prometheus-sql-exporter |     5
 sentry   | prometheus-sql-exporter |     5
          |                         |     5
 grafana  | prometheus-sql-exporter |     5
 keycloak | prometheus-sql-exporter |     5
 postgres | prometheus-sql-exporter |    20
 keycloak | PostgreSQL JDBC Driver  |     9
 netbox   |                         |     7
(12 rows)

closed

mentioned in commit swh/infra/puppet/puppet-swh-site@a6a3220d

Max connections bumped to 400...