[Hardware] Install the metal03 production compute node

changed milestone to %Dynamic infrastructure [Roadmap - Tooling and infrastructure]

added hardware kubernetes labels

assigned to @vsellier

Taking the task to define the installation information

changed the description

marked the checklist item Declare the servers in the inventory as completed

changed the description

made the issue confidential

made the issue visible to everyone

changed the description

The server should be installed the 2023-09-06

The server is racked and up.

The iLo is responding. The credentials are in the credential store.

great thx, we'll install it tomorrow then (w/ @guillaume).

marked the checklist item Add the management info in the credential store as completed

unassigned @vsellier

changed the description

mentioned in commit ipxe@9c92ba70

changed the description

(w/ gsamson)

Build configuration and push them to pergamon:

$ cat configs/rancher-node-metal03.yaml
---
VLAN_ID: 440
IPADDRESS: 192.168.100.133
NETMASK: 255.255.255.0
GATEWAY: 192.168.100.1
NAMESERVER: 192.168.100.29
HOSTNAME: rancher-node-metal03
DOMAINNAME: internal.softwareheritage.org
DEPLOYMENT: production
SUBNET: sesi_rocquencourt
BOOT_DISK_ID_PATTERN: "*_Boot_Controller_*"%
$ HOSTNAME=rancher-node-metal03
./configs/build_iso.sh $HOSTNAME

make: Entering directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
  [DEPS] image/embedded.c
  [BUILD] bin-x86_64-efi/embedded.o
  [VERSION] bin-x86_64-efi/version.ipxe.efi.o
  [AR] bin-x86_64-efi/blib.a
ar: creating bin-x86_64-efi/blib.a
  [LD] bin-x86_64-efi/ipxe.efi.tmp
  [FINISH] bin-x86_64-efi/ipxe.efi
  [GENFSIMG] bin-x86_64-efi/ipxe.iso
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

Makefile.efi:54: warning: pattern recipe did not update peer target 'bin-x86_64-efi/ipxe.usb'.
rm bin-x86_64-efi/version.ipxe.efi.o bin-x86_64-efi/ipxe.efi
make: Leaving directory '/home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/src'
built iso image /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/rancher-node-metal03.iso
Generated preseeding config in /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/rancher-node-metal03.txt and /home/tony/work/inria/repo/swh/sysadm-environment/swh-ipxe/configs/preseeding/finish_install/rancher-node-metal03.sh.

$ rsync -av --include="*/" --include="${HOSTNAME}.sh" --include="${HOSTNAME}.txt" \
--exclude="*" configs/preseeding/ \
pergamon.internal.softwareheritage.org:/srv/softwareheritage/preseeding/
sending incremental file list
rsync: failed to set times on "/srv/softwareheritage/preseeding/.": Operation not permitted (1)
./
rancher-node-metal03.txt
rsync: failed to set times on "/srv/softwareheritage/preseeding/finish_install": Operation not permitted (1)
finish_install/
finish_install/rancher-node-metal03.sh

sent 20,693 bytes  received 282 bytes  41,950.00 bytes/sec
total size is 20,412  speedup is 0.97
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]
$ ssh pergamon.internal.softwareheritage.org ls -lah /srv/softwareheritage/preseeding/rancher-node-metal03.txt /srv/softwareheritage/preseeding/finish_install/rancher-node-metal03.sh
-rw-r--r-- 1 ardumont ardumont 1.8K Sep  8 08:29 /srv/softwareheritage/preseeding/finish_install/rancher-node-metal03.sh
-rw-r--r-- 1 ardumont sudo      19K Sep  8 08:29 /srv/softwareheritage/preseeding/rancher-node-metal03.txt

changed the description

mentioned in commit swh/infra/puppet/puppet-swh-site@39c6630b

marked the checklist item Add puppet configuration as completed

First run of puppet was ran (and failed [1]). Which allowed to have the record registered in pergamon (puppet agent -t there). Which allowed to install the proper source.list to install zfs-dkms. Install of zfs-dkms, then reboot. A second puppet agent -t went through.

And the zfs configuration went through:

root@rancher-node-metal03:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
data          1.14M  3.36T       96K  /data
data/docker    172K  3.36T      172K  /var/lib/docker
data/kubelet    96K  3.36T       96K  /var/lib/kubelet
data/rancher    96K  3.36T       96K  /var/lib/rancher

Remains to:

Configure the data/swap & data/volumes to create.
Register rancher-node-metal03 in the production cluster
Add labels to the nodes so pods can be scheduled there.

[1] Unexpectedly expected ;). The puppet manifests expect the zfs-dkms to be installed. Which is true for most of our machines (especially vms) but not for bare machines.

marked the checklist item Run puppet on the node as completed

changed the description

Swap:

root@rancher-node-metal03:~# mkswap -L swap /dev/zvol/data/swap
Setting up swapspace version 1, size = 256 GiB (274877902848 bytes)
LABEL=swap, UUID=20fffd00-7477-4053-a2d6-bb2b709197a7
root@rancher-node-metal03:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
data           264G  3.10T       96K  /data
data/docker    172K  3.10T      172K  /var/lib/docker
data/kubelet    96K  3.10T       96K  /var/lib/kubelet
data/rancher    96K  3.10T       96K  /var/lib/rancher
data/swap      264G  3.36T       64K  -
root@rancher-node-metal03:~# grep swap /etc/fstab
#/dev/mapper/rancher--node--metal03--vg-swap_1  none    swap    sw      0       0
LABEL="swap"    swap    swap    sw      0       0
root@rancher-node-metal03:~# swapon /dev/zvol/data/swap

marked the checklist item swap file in data/swap (256Go) as completed

marked the checklist item zfs needs to be configure with a couple of datasets (-> puppet does most of it) as completed

Ensure /tmp is a tmpfs of >= 256Go (/etc/fstab)

Another puppet agent -t and it created the /tmp as wanted.

root@rancher-node-metal03:~# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for rancher-node-metal03.internal.softwareheritage.org
Info: Applying configuration version '1694175712'
Notice: /Stage[main]/Profile::Mountpoints/Mount[/tmp]/options: options changed 'size=1023406080,nr_inodes=200m,noexec,n
osuid,nodev,relatime,rw' to 'size=275901308928,nr_inodes=200m,noexec,nosuid,nodev,relatime,rw'
Info: Computing checksum on file /etc/fstab
Info: /Stage[main]/Profile::Mountpoints/Mount[/tmp]: Scheduling refresh of Mount[/tmp]
Info: Mount[/tmp](provider=parsed): Remounting
Notice: /Stage[main]/Profile::Mountpoints/Mount[/tmp]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Mountpoints/Mount[/tmp]: Scheduling refresh of Mount[/tmp]
Notice: Applied catalog in 17.24 seconds