Deploy the journal scrubber in production

After #5108 (closed), the journal scrubbers can be deployed in production

the deployment plan is the same as for staging

changed milestone to %Regularly scrub journal, storage, and objstorage [Roadmap - Preserve]

assigned to @vsellier

mentioned in commit swh/infra/ci-cd/swh-charts@45d1f9af

mentioned in merge request swh/infra/ci-cd/swh-charts!209 (merged)

In the meantime, the static scrubbers were stopped.

mentioned in commit swh/infra/ci-cd/swh-charts@a8d35817

• old postgresql database renamed:

root@belvedere:~# sudo -i -u postgres psql -p 5434
psql (12.15 (Debian 12.15-1.pgdg110+1))
Type "help" for help.
postgres=# alter database "swh-scrubber" rename to "swh-scrubber-backup-5139";

ALTER DATABASE

root@belvedere:~# sudo -i -u postgres createdb -p 5434 --lc-ctype=C.UTF-8 -T template1 -O swh-scrubber swh-scrubber

^ just as in #4324 (closed)

mentioned in commit swh/infra/ci-cd/swh-charts@30fc65b6

• Database configuration:

swh@swh-toolbox-8499ff5477-ghgpp:~$ export SWH_CONFIG_FILENAME=/etc/swh/config-scrubber-journal.yml
swh@swh-toolbox-8499ff5477-ghgpp:~$ swh db init scrubber
psql:/opt/swh/.local/lib/python3.10/site-packages/swh/core/db/sql/36-dbmodule.sql:12: NOTICE:  relation "dbmodule" already exists, skipping
INFO:swh.core.cli.db:Initializing database version to 7 from the scrubber datastore
DONE database for scrubber initialized at version 7

swh-scrubber=> \d
                      List of relations
 Schema |           Name           |   Type   |    Owner
--------+--------------------------+----------+--------------
 public | check_config             | table    | swh-scrubber
 public | check_config_id_seq      | sequence | swh-scrubber
 public | checked_partition        | table    | swh-scrubber
 public | corrupt_object           | table    | swh-scrubber
 public | datastore                | table    | swh-scrubber
 public | datastore_id_seq         | sequence | swh-scrubber
 public | dbmodule                 | table    | swh-scrubber
 public | dbversion                | table    | swh-scrubber
 public | fixed_object             | table    | swh-scrubber
 public | missing_object           | table    | swh-scrubber
 public | missing_object_reference | table    | swh-scrubber
 public | object_origin            | table    | swh-scrubber
(12 rows)

• checker configuration:

swh@swh-toolbox-8499ff5477-ghgpp:~$ for type in directory snapshot release revision
do
  swh scrubber check init journal --object-type=$type --name journal-checker-$type
done
Created configuration journal-checker-directory [1] for checking directory in kafka journal
Created configuration journal-checker-snapshot [2] for checking snapshot in kafka journal
Created configuration journal-checker-release [3] for checking release in kafka journal
Created configuration journal-checker-revision [4] for checking revision in kafka journal

swh-scrubber=> \pset pager off
Pager usage is off.
swh-scrubber=> select * from datastore;
 id | package | class |                                                                                                                                     instance                                                                           
----+---------+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  1 | journal | kafka | {"brokers": ["kafka1.internal.softwareheritage.org:9094", "kafka2.internal.softwareheritage.org:9094", "kafka3.internal.softwareheritage.org:9094", "kafka4.internal.softwareheritage.org:9094"], "group_id": "swh-archive-prod-journalchecker", "prefix": "swh.journal.objects"}
(1 row)

Time: 5.210 ms
swh-scrubber=> select * from check_config;
 id | datastore | object_type | nb_partitions | check_hashes | check_references |           name            | comment
----+-----------+-------------+---------------+--------------+------------------+---------------------------+---------
  1 |         1 | directory   |          4096 | t            | f                | journal-checker-directory |
  2 |         1 | snapshot    |          4096 | t            | f                | journal-checker-snapshot  |
  3 |         1 | release     |          4096 | t            | f                | journal-checker-release   |
  4 |         1 | revision    |          4096 | t            | f                | journal-checker-revision  |
(4 rows)

Time: 7.947 ms

• kube nodes labelled:

 seq 1 3 | xargs -n1 -i{} -t kubectl --context archive-production-rke2 label nodes rancher-node-metal0{} swh/scrubber=true                                                                                                        12:05:57
xargs: warning: options --max-args and --replace/-I/-i are mutually exclusive, ignoring previous --max-args value
kubectl --context archive-production-rke2 label nodes rancher-node-metal01 'swh/scrubber=true'
node/rancher-node-metal01 labeled
kubectl --context archive-production-rke2 label nodes rancher-node-metal02 'swh/scrubber=true'
node/rancher-node-metal02 labeled
kubectl --context archive-production-rke2 label nodes rancher-node-metal03 'swh/scrubber=true'
node/rancher-node-metal03 labeled

First corrupted object occurrences are detected:

swh-scrubber=> select config_id, count(id) from corrupt_object group by config_id;
 config_id | count
-----------+-------
         4 |     9
         3 |  2041
(2 rows)

Time: 7.159 ms

marked this issue as related to swh/devel/swh-scrubber#4698 (closed)

mentioned in commit swh/infra/ci-cd/swh-charts@99378712

mentioned in commit swh/infra/ci-cd/swh-charts@b5f98367

The directory checker is disabled until a fix or workaround is available for swh/devel/swh-scrubber#4698 (closed)

marked this issue as related to swh/devel/swh-scrubber#4699

removed the relation with swh/devel/swh-scrubber#4699

closed

For the record, the old database was never accessed since one year. It's time to remove it to recover the space we can.