Upgrade rancher's kubernetes clusters to a supported version (>=1.25)

1.23/1.24 versions are not supported since the 2023-07-28 (https://kubernetes.io/releases/)

After the rancher's upgrade, the 1.23, 1.24, 1.25, 1.26 and 1.27 version are usable.

Gitlab is still not officially supporting 1.25+ version

• rancher-cluster is already in 1.26 (#4998 (closed))
• gitlab staging need to wait for [1]
• gitlab production need to wait for [1]
• admin cluster
• archive-staging-rke2 cluster
• archive-production-rke cluster

[1] https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/issues/1048

The admin cluster should be upgradable from 1.23 to 1.24 because the performance issues are resolved.

The 1.24 to 1.25+ upgrade will be treated in separate task because several component needs to be upgraded before (to remove the usage of the PodSecurityPolicy object)

changed milestone to %MRO 2023

added kubernetes rancher labels

assigned to @vsellier

marked this issue as related to #4688 (closed)

added #5000 (closed) as child task

changed the description

removed child task #5000 (closed)

admin cluster upgraded from 1.23 to 1.24.

There was some incompatibility with the oldest version of docker and the new rancher agent version.

Docker needed to be upgraded when the node was drained by rancher. I took the opportunity to perform the pending kernal upgrades and reboot the vms.

marked this issue as related to #4998 (closed)

Admin cluster PSPs:

kubectl --context cluster-admin get podsecuritypolicies                                            14:52:00
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME                                                                  PRIV    CAPS   SELINUX    RUNASUSER          FSGROUP     SUPGROUP    READONLYROOTFS   VOLUMES
admin-blackbox-exporter-application-prometheus-blackbox-exporte-psp   false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,secret
admin-cluster-components-blackbox-exporter-psp                        false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,secret
rancher-monitoring-alertmanager                                       false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-crd-manager                                        false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,secret
rancher-monitoring-grafana                                            false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,csi,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-kube-state-metrics                                 false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            secret
rancher-monitoring-operator                                           false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-patch-sa                                           false          RunAsAny   MustRunAsNonRoot   MustRunAs   MustRunAs   false            secret
rancher-monitoring-prometheus                                         false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
rancher-monitoring-prometheus-adapter                                 false          RunAsAny   MustRunAs          RunAsAny    RunAsAny    false            secret,emptyDir,configMap
rancher-monitoring-prometheus-node-exporter                           false          RunAsAny   RunAsAny           MustRunAs   MustRunAs   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath

• admin-blackbox-exporter-application-prometheus-blackbox-exporte-psp is a PSP not cleaned up after the removal of the admin-blackbox-exporter-application argocd application
• admin-cluster-components-blackbox-exporter-psp is created by blackbox exporter
• all the monitoring ones are related to the rancher prometheus deployement

mentioned in commit swh/infra/ci-cd/swh-charts@87939f3f

mentioned in commit swh-sysadmin-provisioning@9b2eeb61

mentioned in commit swh-sysadmin-provisioning@c526c653

mentioned in commit swh-sysadmin-provisioning@fdffbd62

The upgrade of the monitoring application to the last 102.0.1 removed the last PodSecurityPolicy objects that were declared on the cluster.

kubectl --context cluster-admin get podsecuritypolicies
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
No resources found

Argocd 2.5 is compatible with kubernetes 1.23, 1.24 and 1.25 (https://argo-cd.readthedocs.io/en/stable/operator-manual/installation/#supported-versions)

It will be possible to upgrade the cluster to 1.25 but argo will need to be upgraded to upgrade to 1.26

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@b304f0b1

marked this issue as related to #4731 (closed)

changed the description

admin cluster upgrade to kubernetes 1.26.7. No issues detected so far

kubectl --context cluster-admin get nodes
NAME                        STATUS   ROLES               AGE    VERSION
rancher-node-admin-mgmt1    Ready    controlplane,etcd   283d   v1.26.7
rancher-node-admin-node01   Ready    worker              283d   v1.26.7
rancher-node-admin-node02   Ready    worker              283d   v1.26.7
rancher-node-admin-node03   Ready    worker              219d   v1.26.7

mentioned in commit swh-sysadmin-provisioning@f6a4dd1c

marked this issue as related to #5014 (closed)

archive-staging-rke2 cluster upgraded to 1.25 and v1.26.7+rke2r1 without services interruption

it seems keda is impacted by the upgrade to 1.26:

2023-08-22T09:54:13Z    ERROR    controller-runtime.source    if kind is a CRD, it should be installed before calling Start    {"kind": "HorizontalPodAutoscaler.autoscaling", "error": "no matches for kind \"HorizontalPodAutoscaler\" in version \"autoscaling/v2beta2\""}
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:139
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:660
k8s.io/apimachinery/pkg/util/wait.poll
    /go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:594
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:545
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.3/pkg/source/source.go:132

This is due to the deprecation of the v2beta2 version of the HorizontalPodAutoscaler

HorizontalPodAutoscaler
La version d'API bêta (autoscaling/v2beta1) de HorizontalPodAutoscaler n'est plus diffusée à partir de la version 1.25. Cette API a été rendue obsolète dans la version 1.23.
Effectuez une migration des fichiers manifestes et des clients API de sorte qu'ils utilisent la version autoscaling/v2 HorizontalPodAutoscaler de l'API.

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@74f9f5e3

mentioned in commit swh-sysadmin-provisioning@151988f7

mentioned in commit swh-sysadmin-provisioning@9c4635df

mentioned in commit swh-sysadmin-provisioning@1b40c146

The production cluster was upgraded by @guillaume to 1.26 without interruption \o/. Thanks

closed

marked the checklist item archive-staging-rke2 cluster as completed

marked the checklist item archive-production-rke cluster as completed

mentioned in commit swh-sysadmin-provisioning@f28b58e7

marked this issue as related to #5021 (closed)