Upgrade rke2 to v1.26.13+rke2r1

A fairly heavy-hitting container escape bug has been found in versions of runc ranging from 1.0.x to 1.1.11; Version 1.1.12 has been released to fix the bug, and k8s 1.26.13, that includes the fix, has been released.

We should perform the upgrade on our clusters.

Since swh-sysadmin-provisioning@3a2549b5 the change should be fairly seamless.

• test cluster
• admin cluster
• staging cluster
• production cluster

added activity::MRO priority:High labels

assigned to @olasd

marked the checklist item test cluster as completed

mentioned in commit swh-sysadmin-provisioning@1ded71de

The upgrade of the test cluster went seamlessly, and no workload pods were restarted; I'll go ahead and roll out the upgrade to the admin cluster now.

Before doing that I'll pause argocd automatic deployments.

mentioned in commit swh-sysadmin-provisioning@8cd1e4d7

marked the checklist item admin cluster as completed

marked the checklist item staging cluster as completed

mentioned in commit swh-sysadmin-provisioning@aba313f6

mentioned in commit swh-sysadmin-provisioning@d1728960

marked the checklist item production cluster as completed

All rke2 clusters have been upgraded.

While doing the upgrade of the production cluster, the slowness of banco has been diagnosed: the KUBE-FIREWALL iptables chain had its rules duplicated around 8k times (generating a total of 16k rules all using the conntrack feature, in both the INPUT and OUTPUT chains). This slowed down all packets to a crawl.

Emptying the chain (and restoring the 2 expected rules) restored normal behavior on banco.

closed