Upgrade gitlabs to 15.7

GitLab 15.7.0 was released on Dec 22. 15.7.1 in January.

Instructions:

• https://docs.gitlab.com/operator/operator_upgrades.html
• https://docs.gitlab.com/operator/gitlab_upgrades.html

ToDo:

• staging: upgrade operator to 0.15.1
• staging: bump chart version to 6.6.3 (GitLab 15.6.3)
• staging: bump chart version to 6.7.1 (GitLab 15.7.1)
• check that staging is ok
• prod: upgrade operator to 0.15.1
• prod: bump chart version to 6.5.7 (GitLab 15.5.6)
• prod: bump chart version to 6.6.3 (GitLab 15.6.3)
• prod: bump chart version to 6.7.1 (GitLab 15.7.1)

added GitLab migration environment: production environment: staging labels

assigned to @olasd

marked the checklist item staging: upgrade operator to 0.14.0 as completed

marked the checklist item staging: bump chart version to 6.5.5 (GitLab 15.5.4) as completed

marked the checklist item staging: bump chart version to 6.6.0 (GitLab 15.6.0) as completed

marked the checklist item check that staging is ok as completed

added state:wip label

Reported our operator silly looping as https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/issues/973. I guess I'll wait for them to give us some more diagnostics hints before doing the prod upgrade to 15.6.x.

changed the description

marked the checklist item staging: bump chart version to 6.6.1 (GitLab 15.6.1) as incomplete

marked the checklist item check that staging is ok as incomplete

We needed to downgrade staging to be able to restore the backup from the production database.

marked the checklist item prod: upgrade operator to 0.14.1 as incomplete

marked the checklist item staging: upgrade operator to 0.14.2 as completed

We had the reconcile loop fun again in staging (no surprises there), and it completed after some hours.

I figured out how to recompile the operator to add some diagnostics:

1. In the gitlab-operator repo, rebuild the image

env IMG_REGISTRY=registry-staging.swh.network IMG_REPOSITORY=swh/infra/sysadm-environment task docker-build

2. push the image

podman push registry-staging.swh.network/swh/infra/sysadm-environment/gitlab-operator:latest

3. add the proper values to the gitlab-operator argocd app

image:
  registry: registry-staging.swh.network
  repository: swh/infra/sysadm-environment
  image: gitlab-operator
  tag: latest

4. redeploy the argocd app with the new values

changed the description

mentioned in commit swh/infra/ci-cd/k8s-clusters-conf@b1cc74b8

marked the checklist item prod: upgrade operator to 0.15.1 as completed

marked the checklist item prod: bump chart version to 6.5.7 (GitLab 15.5.6) as completed

changed title from Upgrade gitlabs to 15.6 to Upgrade gitlabs to 15.7

marked the checklist item staging: bump chart version to 6.7.1 (GitLab 15.7.1) as completed

marked the checklist item check that staging is ok as completed

marked the checklist item prod: bump chart version to 6.6.3 (GitLab 15.6.3) as completed

marked the checklist item prod: bump chart version to 6.7.1 (GitLab 15.7.1) as completed

closed