Upgrade GitLab to 16.x (helm chart to 7.x)
gitlab.softwareheritage.org is currently running GitLab 15.11.x (chart 6.11.x), GitLab is now at 16.1.1 (chart 7.1.1, two monthly releases further).
The upgrade to GitLab 16.x comes with an upgrade of the following components:
Both of these upgrades need the frontend components to be completely shut down, and the postgresql upgrade needs a dump/restore, so some downtime (around 1h) needs to be planned.
The upgrade procedure, which was tested on the staging instance (freshly reset to a backup of the production instance), looks like the following:
- upgrade GitLab operator to latest version (in k8s-clusters-config)
- upgrade GitLab to latest available 15.11.x (in k8s-private-data)
- retrieve PostgreSQL database upgrade script locally
- disable self-healing of GitLab-related resources in argocd (k8s-clusters-config)
-
update the GitLab custom resource with the latest available 7.0.x chart version and migrations disabled. immediately scale down operator deployment to avoid applying the new template
kubectl edit gitlab gitlab && kubectl scale deployment --replicas 0 --selector 'control-plane=controller-manager'
- --- BEGIN long downtime
-
scale down front-end service deployments to avoid changes during database upgrade
kubectl scale deployment --replicas 0 --selector 'app in (webservice, sidekiq, kas, gitlab-exporter)'
-
database upgrade steps: dump database using the pre-upgrade script
./database-upgrade pre
-
database upgrade steps: drop the postgresql pvc and statefulset
kubectl delete statefulset gitlab-postgresql && kubectl delete pvc data-gitlab-postgresql-0
-
redis chart upgrade steps: drop the redis statefulset
kubectl delete statefulset gitlab-redis-master
-
scale up operator deployment. this will apply the chart upgrade to 7.0.x
kubectl scale deployment --replicas 1 --selector 'control-plane=controller-manager'
-
when toolbox container is ready, scale down operator deployment again (to avoid reconciles while the db upgrade script is running)
- wait for toolbox to be deployed with the new chart version:
while ! test $(kubectl get deployment/gitlab-toolbox -o 'jsonpath={.spec.template.metadata.labels.chart}) = 'toolbox-7.0.x'; do sleep 10; done; kubectl rollout status -w deployment/gitlab-toolbox; kubectl scale deployment --replicas 0 --selector 'control-plane=controller-manager'
- wait for toolbox to be deployed with the new chart version:
-
database upgrade steps: run post upgrade script
./database-upgrade post
-
scale operator deployment back up, wait for GitLab 16.0.x to finish reconciling
kubectl scale deployment --replicas 1 --selector 'control-plane=controller-manager' && kubectl wait --for condition=Available=true gitlab/gitlab
- --- END long downtime, GitLab is at 16.0.x
- restore self-healing of GitLab-related resources in ArgoCD
- update GitLab custom resource to chart 7.1.x, with migrations enabled
- wait for operator to reconcile GitLab 16.1.x
Activity
changed milestone to %MRO 2023
assigned to @olasd
mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@c75c589f
marked the checklist item retrieve PostgreSQL database upgrade script locally as completed
mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@7b98cbe5
A few caveats noticed during the upgrade:
- kubernetes decided to downscale the cluster during the upgrade when all the frontend service pods were down. Unfortunately this evicted the postgresql pod, in the middle of the db upgrade, so that had to be done again.
- either find a way to mark the postgresql pod as harder to evict (how?)
- or disable cluster autoscaling during such upgrades (how?)
- or both
- looks like the container registry storage backed by azure fixed a bug where the default storage had a double slash in the path, so the new version of the registry couldn't find any data
- resolved by using azcopy to move the files to the new path
- kubernetes decided to downscale the cluster during the upgrade when all the frontend service pods were down. Unfortunately this evicted the postgresql pod, in the middle of the db upgrade, so that had to be done again.
looks like the container registry storage backed by azure fixed a bug where the default storage had a double slash in the path, so the new version of the registry couldn't find any data
That might explain why i had issues at some point in my local minikube where some images could not be pulled from the gitlab registry (iiyr). And then it's now subsided.
resolved by using azcopy to move the files to the new path
So thanks for everything (and the fix too ;).
either find a way to mark the postgresql pod as harder to evict (how?)
Not sure it's easily doable, if the node is drained, the pod will be eventually killed whatever the configuration is
or disable cluster autoscaling during such upgrades (how?)
This one is easier, you can increase the minimal pod count of the pool so it will not be downscaled. It can be done via terraform https://gitlab.softwareheritage.org/swh/infra/swh-sysadmin-provisioning/-/blob/master/azure/terraform/gitlab.tf#L15 or directly in the ui.
It's also possible to temporary configure the pool to manual scaling
kubernets cluster > node pool > default > scale
So thanks for everything (and the fix too ;).
indeed
-
The container registry copy was slightly incomplete: 50 files out of 3k had failed to copy completely.
As I had generated a SAS without delete permissions, azcopy couldn't do an automatic retry, nor delete the files, so some image pulls were corrupted (e.g. the latest lister image)
I manually removed the broken files in the destination path and resynced the data, the pulls now succeed.
