[rancher-staging-rke2] Prepare the upgrade to kubernetes 1.25+

There is a couple of PodSecurityPolicies to remove:

kubectl --context archive-staging-rke2 get podsecuritypolicies
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME                                          PRIV    CAPS           SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
calico-kube-controllers                       false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
calico-node                                   true                   RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
calico-typha                                  false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
global-restricted-psp                         false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
global-unrestricted-psp                       true    *              RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
rancher-monitoring-alertmanager               false                  RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-crd-manager                false                  RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,secret
rancher-monitoring-grafana                    false                  RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,csi,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-kube-state-metrics         false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            secret
rancher-monitoring-operator                   false                  RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-patch-sa                   false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            secret
rancher-monitoring-prometheus                 false                  RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-prometheus-adapter         false                  RunAsAny   MustRunAs          RunAsAny    RunAsAny    false            secret,emptyDir,configMap
rancher-monitoring-prometheus-node-exporter   false                  RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
system-unrestricted-psp                       true    *              RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
system-upgrade-controller                     true    CAP_SYS_BOOT   RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
tigera-operator                               false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            hostPath,configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

Most of them are created by the rancher monitoring application, the others seems to be created by rancher components (calico, tigera, ...)

changed milestone to %MRO 2023

added kubernetes rancher labels

assigned to @vsellier

mentioned in commit swh-sysadmin-provisioning@17b7e871

After the upgrade of the monitoring app, the number of PSPs was reduced to 8 (previously 17)

An upgrade from 1.24.9 to 1.24.16 to test if the removal of the PSPs were prepared but it seems not as all the PSPs are still presents.

During the upgrade, rancher drained the node, upgraded and restarted each node without delay between each.

The pods didn't had the time to restart in the interval, and finally most of the pods were down at the end of the upgrade.

The cluster took a long time (~30mn) to finally stabilize. It generated some downtime on the exposed services like graphql, cassandra webapp. It's should not happen.

Someone asked the question here: https://github.com/rancher/system-upgrade-controller/discussions/177. The solution looks not so easy. Perhaps we should test without draining the nodes.

A test cluster was created in order to test the migration to 1.25+ and how to avoid the downtime during the cluster rolling upgrade.

After a fresh install with 1.24+ version, the PSPs are presents just as in the staging cluster:

> kubectl get podsecuritypolicies
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME                        PRIV    CAPS           SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
calico-kube-controllers     false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,downwardAPI,emptyDir,persistentVolumeClaim,projected,secret
calico-node                 true                   RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,downwardAPI,emptyDir,persistentVolumeClaim,projected,secret,hostPath
calico-typha                false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,downwardAPI,emptyDir,persistentVolumeClaim,projected,secret
global-restricted-psp       false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
global-unrestricted-psp     true    *              RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
system-unrestricted-psp     true    *              RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
system-upgrade-controller   true    CAP_SYS_BOOT   RunAsAny   RunAsAny           RunAsAny    RunAsAny    false            *
tigera-operator             false                  RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            hostPath,configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim

With the test cluster, we have tested the upgrade to 1.25 and 1.26 with the drain node option disabled. Rancher upgrades the cluster without killing the running pods and without stopping the services.

The PSPs deployed are removed and it seems there is no issue with the further migration.

So the staging cluster should be upgradable to 1.26 if the drain node option is disabled.

mentioned in commit swh-sysadmin-provisioning@6f8e2485

closed

marked this issue as related to #4999 (closed)

mentioned in commit swh-sysadmin-provisioning@19fa537f

mentioned in commit swh-sysadmin-provisioning@3a2549b5