Document the gitlab installation and procedures

Related to infra/sysadm-environment#4647

sysadm/support-services/gitlab/index.rst

+.. _gitlab:
+
+Gitlab
+======
+
+.. toctree::
+   :maxdepth: 1
+
+   installation
+

sysadm/support-services/gitlab/installation.rst

+.. _gitlab_installation:
+
+.. admonition:: Intended audience
+   :class: important
+
+   sysadm staff members
+
+
+Gitlab installation
+===================
+
+The deployment of the gitlab instances is managed by:
+  - terraform for the the infrastructure part
+  - ArgoCd for the Gitlab deployment
+
+Each deployment relies on a dedicated `AKS` instance and a dedicated blob storage.
+
+Infrastructure
+--------------
+
+The `terraform code <https://gitlab.softwareheritage.org/infra/swh-sysadmin-provisioning/-/tree/master/azure/terraform/modules/gitlab>`__ is deploying
+and configuring a full, dedicated Azure Kubernetes Service matching this environment:
+
+.. figure:: ../../images/gitlab-infrastructure.png
+   :alt: Gitlab Infrastructure
+
+(`source <https://archive.softwareheritage.org/swh:1:dir:713a9ee3a2191aa2a5f386798442bc321c0b4747;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:7336142b5e1a03be4eedfa3f9edc4552297abd82;anchor=swh:1:rev:3045f3c9cca4ff0611725bae143c0ccc5027ed4c;path=/sysadmin/docs/gitlab/>`__)
+
+The output of terraform displays all the necessary information to connect to the kubernetes.
+It's also possible to get the information by executing those commands:
+
+.. code:: bash
+
+    terraform output gitlab-<instance>_aks_summary
+    terraform output --raw gitlab-<instance>_storage_summary
+
+Once the infrastructure is created and available, Gitlab is deployed by ArgoCD.
+It's done by deploying the `Gitlab Operator <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator>`__
+and a gitlab configuration (for example, the
+`staging configuration <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-swh-private-data/-/blob/master/gitlab-staging/gitlab-staging.yaml>`__ ).
+
+A couple of other components are also deployed in parallel to manage the monitoring, an additional ingress controller and cert-manager.
+The applications are available in the `k8s-clusters-conf repository <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/tree/master/argocd/applications/gitlab-staging>`__.
+
+Important components
+~~~~~~~~~~~~~~~~~~~~
+
+- The gitlab transactional data is stored on a couple of kubernetes PVs associated to their relative Azure volumes
+
+.. code:: bash
+
+    kubectl -n gitlab-system get pv                                                                                                                                                                                                                                  17:36:19
+    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                               STORAGECLASS   REASON   AGE
+    pvc-38afb357-36e4-4661-9be6-011f58a24c95   64Gi       RWO            Retain           Bound    gitlab-system/repo-data-gitlab-gitaly-0                                                                             default                 20d
+    pvc-46544290-0278-4b41-b616-7466cd5140f9   8Gi        RWO            Retain           Bound    gitlab-system/data-gitlab-postgresql-0                                                                              default                 20d
+    pvc-988b8047-2ac9-426b-8792-37d749141657   8Gi        RWO            Retain           Bound    gitlab-system/redis-data-gitlab-redis-master-0                                                                      default                 20d
+    pvc-c55bdefa-089b-4cda-b0d0-a1daafb6f37f   16Gi       RWO            Retain           Bound    monitoring/prometheus-gitlab-production-promethe-prometheus-db-prometheus-gitlab-production-promethe-prometheus-0   default                 19d
+
+By default, the operator use a `Delete` reclaim policy which is risky. After the initial installation, it was manually changed to switch to a `Retain` policy
+
+- The gitlab assets (attachments, container registry data, ...) are stored in a dedicated object storage account
+
+Backups
+-------
+
+<Not yet available>
+
+.. note:: The backup will be stored on a minio instance installed on the Rocquencourt infrastructure.
+    see `#4645 <https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4645>`__
+
+Upgrades
+--------
+
+An upgrade of gitlab is done in two major steps (**always test in staging before upgrading the production environment**)
+
+It's important to follow the operatore releases to not be more than 3 minor versions late, or the migration will not be supported be the operator.
+
+- Step 1 `Upgrade the operator <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/operator_upgrades.md>`__
+
+    - Check the last `gitlab operator release <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/releases>`__ and the chart versions mapping
+    - Update the operator version in the `application declaration <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/blob/master/argocd/applications/gitlab-staging/gitlab-operator.yaml>`__. `Example of upgrade <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/commit/1a349433d45623128deabe7ca4ca57b740e16cba>`__
+    - Commit and push the change.
+    - Wait a couple of minutes to let ArgoCD applies the change
+    - A restart of the operator pod should happen (check the gitlab-controller-manager pod).
+    - Something like this should be visible in the operator logs:
+
+.. code:: shell
+
+    > kubernetes logs gitlab-controller-manager-xxxx-xxxx
+    Configuration error detected: chart version 6.5.1 not supported; please use one of the following: 6.5.2, 6.4.4, 6.3.5
+
+At this point, The gitlab operator is updated but the Gitlab upgrade itself is not yet started.
+
+- Step 2 `Upgrade the gitlab version <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/gitlab_upgrades.md>`__
+
+    - Follow the recommended upgrade path depending of the current deployed version of the gitlab instance and the allowed versions of the operator
+    - Update the chart version matching the desired gitlab version (should be present on the release page of the operator) in the
+      `gitlab resource declaration <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-swh-private-data/-/blob/master/gitlab-staging/gitlab-staging.yaml>`__
+    - Commit and push the change
+    - Wait a couple of minutes to let ArgoCD applies the change
+    - A upgrade of the gitlab services should happen and the launch of a migration pod should be done
+    - After a couple of minutes (could be as long as an hour), the cluster should stabilize if everything is ok (no pending pods, no pods in error, ...)
+    - Check the deployed version and the operator logs to check if nothing is wrong
+
+.. code:: shell
+
+    > kubectl -n gitlab-system get gitlab                                                                                                                                                                                                                          17:33:39
+    NAME     STATUS    VERSION
+    gitlab   Running   6.5.2

sysadm/support-services/index.rst

 Support services
 ================
 
+.. toctree::
+   :maxdepth: 2
 
-
-.. todo::
-   This page is a work in progress.
+   gitlab/index