production: Test swh.graph workload in kubernetes

requisite:

• graph must run on rancher-node-metal05 (old "met" with sufficient memory and disks)

plan:

• Add swh/graph=true label to rancher-node-metal05
• Transfer graph dataset from maxxi to rancher-node-metal05 (zfs snapshot)
• Reference graph.i.s.o & graph-grpc.i.s.o as CNAME (for internal use purposes)
• swh/infra/ci-cd/swh-charts!486 (merged): swh-charts: Add values to start the graph deployment in kube (only prepare the volumes and start nothing)
• Configure kubernetes volumes
  • Declare in the chart the persistent volume (and start the service so proper pv gets created)
  • Manually mount the zfs dataset in that persistent volume's path mounted in the host machine
• swh-charts: Incrementally start the deployment (prepareMemoryVolume: true, startService: true)
• Allow reindexation to happen from the chart
• Investigate further issue ~> old dataset used
• Transfer most recent dataset from granet
• Deploy new graph using the most recent dataset (1 grpc, 1 rpc)
• Checks
  • grpc? ok
  • rpc? ok
• Test graph api through webapp-postgresql instance
• monitoring
  • grafana board
    • workload: https://grafana.softwareheritage.org/goto/wgG5ONiHz?orgId=1
    • nginx: https://grafana.softwareheritage.org/goto/xt9eONmNz?orgId=1
  • Add blackbox probe
• go for production deployment

Cleanup:

• Clean up old and wrong dataset from the first tryout

changed the description

changed the description

changed the description

• Transfer graph dataset from maxxi to rancher-node-metal05 (zfs snapshot)

The rancher-node-metal05's root ssh access to maxxi has been added manually (keypair generated, public key added to maxxi root's authorized_keys). Then a snapshot got created on maxxi. Then, a destination dataset has been prepared on rancher-node-metal05. And finally, the actuall transfert happened.

[1]

root@maxxi:~# zfs snapshot ssd/datasets/2023-09-06/compressed@graph-2023-09-06
root@maxxi:~# zfs list -t snapshot
NAME                                                  USED  AVAIL     REFER  MOUNTPOINT
ssd/datasets/2023-08-07@test                            0B      -      368K  -
ssd/datasets/2023-09-06/compressed@graph-2023-09-06     0B      -     4.04T  -

[2]

# Dataset creation (next time, set the compression here)
root@rancher-node-metal05:~# zfs create data/datasets/2023-09-06
# List of datasets
root@rancher-node-metal05:~# zfs list | grep -v rancher
NAME                                  USED  AVAIL     REFER  MOUNTPOINT
data                                 9.52T  22.0T       96K  /data
data/datasets                        8.83T  22.0T      112K  /data/datasets
data/datasets/2023-08-07              424K  22.0T      368K  /data/datasets/2023-08-07
data/datasets/2023-09-06             8.83T  22.0T      104K  /data/datasets/2023-09-06
data/datasets/2023-09-06/compressed  8.83T  22.0T     8.83T  /data/datasets/2023-09-06/compressed
data/kubelet                         99.3M  22.0T     99.3M  /var/lib/kubelet
data/swap                             528G  22.1T      517G  -
data/volumes                           96K  22.0T       96K  /srv/kubernetes/volumes

# Actual transfer of the snapshot (root ssh access to maxxi got added manually first)
root@rancher-node-metal05:~# time ssh root@maxxi zfs send ssd/datasets/2023-09-06/compressed@graph-2023-0
9-06 | zfs receive data/datasets/2023-09-06/compressed

real    301m46.575s
user    170m43.448s
sys     199m48.582s

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@0dc85af3

changed the description

[x] Reference graph.i.s.o & graph-grpc.i.s.o as CNAME (for internal use purposes)

root@pergamon:~# host graph.internal.softwareheritage.org
graph.internal.softwareheritage.org is an alias for k8s-archive-production-rke2.internal.softwareheritage.org.
k8s-archive-production-rke2.internal.softwareheritage.org has address 192.168.100.139
(swh-e2e-checks) root@pergamon:~# host graph-grpc.internal.softwareheritage.org
graph-grpc.internal.softwareheritage.org is an alias for k8s-archive-production-rke2.internal.softwareheritage.org.
k8s-archive-production-rke2.internal.softwareheritage.org has address 192.168.100.139

mentioned in commit swh/infra/ci-cd/swh-charts@ce7eb26e

mentioned in merge request swh/infra/ci-cd/swh-charts!486 (merged)

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@0088126a

I've changed the CNAME to graph-rpc as graph was already an alias for granet.

root@pergamon:~# host graph.internal.softwareheritage.org
graph.internal.softwareheritage.org is an alias for granet.internal.softwareheritage.org.
granet.internal.softwareheritage.org has address 192.168.100.51
root@pergamon:~# host graph-rpc.internal.softwareheritage.org
graph-rpc.internal.softwareheritage.org is an alias for k8s-archive-production-rke2.internal.softwareheritage.org.
k8s-archive-production-rke2.internal.softwareheritage.org has address 192.168.100.139

mentioned in commit swh/infra/ci-cd/swh-charts@7349b277

mentioned in commit swh/infra/ci-cd/swh-charts@8ee301c3

mentioned in commit swh/infra/ci-cd/swh-charts@fc3dbff8

mentioned in commit swh/infra/ci-cd/swh-charts@58d16f31

mentioned in commit swh/infra/ci-cd/swh-charts@f29899d6

mentioned in commit swh/infra/ci-cd/swh-charts@85a306ba