Integrate granet (as rancher-node-highmem01) in production rancher cluster

To get some redundancy in the swh-graph deployment, we want to integrate granet in the rancher cluster as a kubernetes execution node.

To distinguish it from other nodes, we're going with rancher-node-highmem01 (and we plan a follow-up rename of rancher-node-metal05 to rancher-node-highmem02).

granet only has a tiny (and full) LVM for its root filesystem so the reinstall will need to happen offline. I don't think preseeding is a sensible option (as I think it'd be better to keep the current partitioning scheme intact, and only wipe the current root partition) so the install will be done by hand.

• allocate new ip address in rancher range in inventory
• update rancher node address range in firewall
• if needed, update puppet to support new host naming pattern
• back up contents of root partition to the zfs hdd pool
• wipe root partition and reinstall debian on it
• decommission granet from puppet
• integrate node to rancher cluster

assigned to @olasd

changed the description

marked the checklist item allocate new ip address in rancher range in inventory as completed

marked the checklist item back up contents of root partition to the zfs hdd pool as completed

marked the checklist item wipe root partition and reinstall debian on it as completed

mentioned in commit swh/infra/puppet/puppet-swh-site@0c048a14

marked the checklist item decommission granet from puppet as completed

marked the checklist item update rancher node address range in firewall as completed

marked the checklist item integrate node to rancher cluster as completed

After boot, a couple of operations have been needed:

• apt install gnupg zfs-dkms puppet
• import zfs pools (using zfs import -f ssd data to rename the ssd pool to the naming scheme of other rancher nodes, and zfs import -f hdd for the hdd pool).
• Fixup the mdadm configuration to remove references to the old SSD raid 0 (which had been detected by the debian installer)
• regenerated the initramfs after the mdadm config changes

As the old SSD raid0 had been started (so mdadm probably wrote some metadata?) I started a scrub on the data zpool.

After a first successful puppet run, I integrated the node in the rancher cluster.

I'll give the zpool scrub some time to finish before rebooting the host and checking whether it comes back up properly.

zfs scrub completed with no issues, and node integrated to k8s cluster successfully.

closed

mentioned in issue #5543 (closed)