Upgrade PVE cluster from 7 to 8

Proxmox VE 7 support ended on 2024-07-31, so we need to upgrade.

https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

The pve7to8 command doesn't flag any particular issues, we just need to be careful to properly handle the "static" virtual machines for clustered systems (particularly rke2 management nodes)

• uffizi
  • all worker nodes of the rancher test cluster
• hypervisor3
  • rancher-node-staging-rke2-mgmt1
  • rancher-node-admin-rke2-node01
  • rancher-node-admin-rke2-mgmt1
  • rancher-node-production-rke2-mgmt3
• pompidou
  • rancher-node-test-rke2-mgmt2
  • all worker nodes of the rancher staging cluster
• mucem
  • rancher-node-test-rke2-mgmt3
  • rancher-node-staging-rke2-mgmt3
  • rancher-node-admin-rke2-node03
  • rancher-node-admin-rke2-mgmt3
  • rancher-node-production-rke2-mgmt1
• branly
  • rancher-node-test-rke2-mgmt1
  • rancher-node-staging-rke2-mgmt2
  • rancher-node-admin-rke2-node02
  • rancher-node-admin-rke2-mgmt2
  • rancher-node-production-rke2-mgmt2

assigned to @olasd

marked this issue as related to #5415 (closed)

marked the checklist item uffizi as completed

uffizi upgraded.

The network interfaces changed names from bullseye to bookworm, so had to fix that up in /etc/network/interfaces before the boot could complete

hypervisor3 upgraded... and not coming back up ._.

ticket submitted for hypervisor3 idrac access, our DSI contact ran out of time today so we'll check back in tomorrow

Looks like idrac access was restored.

Even though I had checked the new version of the persistent interface naming, and the name was not supposed to change... it changed anyway! Updating /etc/network/interfaces with the new names restored connectivity.

hypervisor3 upgrade complete.

marked the checklist item hypervisor3 as completed

marked the checklist item mucem as completed

Upgraded mucem. No spurious interface name changes this time.

changed the description

marked the checklist item branly as completed

Upgraded branly. network interface name changes: eno33 -> eno33np0, eno34 -> eno34np1

marked the checklist item pompidou as completed

pompidou was upgraded, completing the cluster upgrade.

Preparing pompidou for a reboot by evicting pods away from the -worker[1-6] nodes created an outage of the staging infra as some kubernetes pods (at least one redis and one swh-web postgresql server) are sticky to the arbitrary worker node where their storage was allocated (on local, scratch storage).

This feels like a somewhat serious regression away from the clearly identified singleton servers we used to have for these data silos, we should make sure that fixing this is on the radar (probably by setting up replication/HA on these services?)

closed