Recycle met as a kubernetes production node

Until a better usage of this server is found, we can recycle it as a production compute node.

The short term usage of met is of course offloading some compute on it but an important mid-term could be to poc the deployment of the graph in the elastic environment.

There is no important data to recover on the server (cc @douardda)

Plan:

• decommission met.internal.softwareheritage.org on pergamon;
• remove met.internal.softwareheritage.org.yaml from the puppet repository;
• update the node's information in the inventory;
• create rancher-node-metal05.internal.softwareheritage.org.yaml in the puppet repository;
• reinstall Debian bullseye with IPXE;
• register the new node in the cluster archive-production-rke2;
• create the kubernetes labels.
• create a swap at least the size of memory (512Go)
• firewall: Reference the ip in the list of the kube workers

added kubernetes label

assigned to @guillaume

changed the description

marked the checklist item decommission met.internal.softwareheritage.org on pergamon; as completed

mentioned in commit swh/infra/puppet/puppet-swh-site@c62b13be

marked the checklist item remove met.internal.softwareheritage.org.yaml from the puppet repository; as completed

marked the checklist item update the node's information in the inventory; as completed

mentioned in commit swh/infra/puppet/puppet-swh-site@cc29d6a6

marked the checklist item create rancher-node-metal05.internal.softwareheritage.org.yaml in the puppet repository; as completed

looks like banco is still trying to backup some met content:

borg@banco.internal.softwareheritage.org:/srv/borg/repositories/met.internal.softwareheritage.org: Error running actions for repository
Command 'borg create --exclude-from /tmp/tmpj4und21m --exclude-caches --exclude-if-present .nobackup borg@banco.internal.softwareheritage.org:/srv/borg/repositories/met.internal.softwareheritage.org::met.internal.softwareheritage.org-{now:%Y-%m-%dT%H:%M:%S.%f} /' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: Error running configuration file

summary:
/etc/borgmatic/config.yaml: Error running configuration file
borg@banco.internal.softwareheritage.org:/srv/borg/repositories/met.internal.softwareheritage.org: Error running actions for repository
Remote: Permission denied, please try again.
Remote: Permission denied, please try again.
Remote: borg@banco.internal.softwareheritage.org: Permission denied (publickey,password).
Connection closed by remote host. Is borg working on the server?
Command 'borg create --exclude-from /tmp/tmpj4und21m --exclude-caches --exclude-if-present .nobackup borg@banco.internal.softwareheritage.org:/srv/borg/repositories/met.internal.softwareheritage.org::met.internal.softwareheritage.org-{now:%Y-%m-%dT%H:%M:%S.%f} /' returned non-zero exit status 2.

marked the checklist item reinstall Debian bullseye with IPXE; as completed

mentioned in commit ipxe@92d10303

mentioned in commit ipxe@1ce32561

marked the checklist item register the new node in the cluster archive-production-rke2. as completed

changed the description

marked the checklist item create the kubernetes labels. as completed

changed the description

• swap created

zfs create -V 512G data/swap
# mkswap -L swap /dev/zvol/data/swap
Setting up swapspace version 1, size = 512 GiB (549755809792 bytes)
LABEL=swap, UUID=a1273fcd-9f74-4d64-972f-ef661e8b66ac

• Updated the initramfs resume configuration:

# cat /etc/initramfs-tools/conf.d/resume
RESUME=none

• old swap partition on the boot disks removed

# lvremove '/dev/rancher-node-metal05-vg/swap_1'
Do you really want to remove active logical volume rancher-node-metal05-vg/swap_1? [y/n]: y
  Logical volume "swap_1" successfully removed

• init ramfs updated

update-initramfs -u

• puppet applied
• server reboot done

closed

marked the checklist item create a swap at least the size of memory (512Go) as completed

changed the description

[x] firewall: Reference the ip in the list of the kube workers

Otherwise, some pods can fail to start (counters for example).