diff --git a/docs/sysadm/support-services/gitlab/installation.rst b/docs/sysadm/support-services/gitlab/installation.rst index 7f576db92568f0702eb8cae36c40fdb428d2b16c..0220f9918b21bf06801b5abb423aceb83dcad071 100644 --- a/docs/sysadm/support-services/gitlab/installation.rst +++ b/docs/sysadm/support-services/gitlab/installation.rst @@ -10,7 +10,8 @@ Gitlab installation =================== The deployment of the gitlab instances is managed by: - - terraform for the the infrastructure part + + - terraform for the infrastructure part - ArgoCd for the Gitlab deployment Each deployment relies on a dedicated `AKS` instance and a dedicated blob storage. @@ -18,34 +19,42 @@ Each deployment relies on a dedicated `AKS` instance and a dedicated blob storag Infrastructure -------------- -The `terraform code <https://gitlab.softwareheritage.org/infra/swh-sysadmin-provisioning/-/tree/master/azure/terraform/modules/gitlab>`__ is deploying -and configuring a full, dedicated Azure Kubernetes Service matching this environment: +The `terraform code +<https://gitlab.softwareheritage.org/infra/swh-sysadmin-provisioning/-/tree/master/azure/terraform/modules/gitlab>`__ +is deploying and configuring a full, dedicated Azure Kubernetes Service matching this +environment: .. figure:: ../../images/gitlab-infrastructure.png :alt: Gitlab Infrastructure (`source <https://archive.softwareheritage.org/swh:1:dir:713a9ee3a2191aa2a5f386798442bc321c0b4747;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:7336142b5e1a03be4eedfa3f9edc4552297abd82;anchor=swh:1:rev:3045f3c9cca4ff0611725bae143c0ccc5027ed4c;path=/sysadmin/docs/gitlab/>`__) -The output of terraform displays all the necessary information to connect to the kubernetes. -It's also possible to get the information by executing those commands: +The output of terraform displays all the necessary information to connect to the +kubernetes cluster. It's also possible to get the information by executing those +commands: .. code:: bash terraform output gitlab-<instance>_aks_summary terraform output --raw gitlab-<instance>_storage_summary -Once the infrastructure is created and available, Gitlab is deployed by ArgoCD. -It's done by deploying the `Gitlab Operator <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator>`__ -and a gitlab configuration (for example, the -`staging configuration <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-swh-private-data/-/blob/master/gitlab-staging/gitlab-staging.yaml>`__ ). +Once the infrastructure is created and available, Gitlab is deployed by ArgoCD. It's +done by deploying the `Gitlab Operator +<https://gitlab.com/gitlab-org/cloud-native/gitlab-operator>`__ and a gitlab +configuration (for example, the `staging configuration +<https://gitlab.softwareheritage.org/infra/ci-cd/k8s-swh-private-data/-/blob/master/gitlab-staging/gitlab-staging.yaml>`__). -A couple of other components are also deployed in parallel to manage the monitoring, an additional ingress controller and cert-manager. -The applications are available in the `k8s-clusters-conf repository <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/tree/master/argocd/applications/gitlab-staging>`__. +A couple of other components are also deployed in parallel to manage the monitoring, an +additional ingress controller and cert-manager. The applications are available in the +`k8s-clusters-conf repository +<https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/tree/master/argocd/applications/gitlab-staging>`__. Important components ~~~~~~~~~~~~~~~~~~~~ -- The gitlab transactional data is stored on a couple of kubernetes PVs associated to their relative Azure volumes +- The gitlab transactional data is stored on a couple of kubernetes PVs associated to + their relative Azure volumes + .. code:: bash @@ -56,9 +65,11 @@ Important components pvc-988b8047-2ac9-426b-8792-37d749141657 8Gi RWO Retain Bound gitlab-system/redis-data-gitlab-redis-master-0 default 20d pvc-c55bdefa-089b-4cda-b0d0-a1daafb6f37f 16Gi RWO Retain Bound monitoring/prometheus-gitlab-production-promethe-prometheus-db-prometheus-gitlab-production-promethe-prometheus-0 default 19d -By default, the operator use a `Delete` reclaim policy which is risky. After the initial installation, it was manually changed to switch to a `Retain` policy +By default, the operator use a `Delete` reclaim policy which is risky. After the initial +installation, it was manually changed to switch to a `Retain` policy -- The gitlab assets (attachments, container registry data, ...) are stored in a dedicated object storage account +- The gitlab assets (attachments, container registry data, ...) are stored in a + dedicated object storage account Backups ------- @@ -71,17 +82,26 @@ Backups Upgrades -------- -An upgrade of gitlab is done in two major steps (**always test in staging before upgrading the production environment**) +An upgrade of gitlab is done in two major steps (**always test in staging before +upgrading the production environment**) -It's important to follow the operatore releases to not be more than 3 minor versions late, or the migration will not be supported be the operator. +It's important to follow the operator releases (at most 3 minor versions late or the +migration will not be supported by the operator). -- Step 1 `Upgrade the operator <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/operator_upgrades.md>`__ +- Step 1 `Upgrade the operator + <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/operator_upgrades.md>`__ - - Check the last `gitlab operator release <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/releases>`__ and the chart versions mapping - - Update the operator version in the `application declaration <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/blob/master/argocd/applications/gitlab-staging/gitlab-operator.yaml>`__. `Example of upgrade <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/commit/1a349433d45623128deabe7ca4ca57b740e16cba>`__ + - Check the last `gitlab operator release + <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/releases>`__ and the + chart versions mapping + - Update the operator version in the `application declaration + <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/blob/master/argocd/applications/gitlab-staging/gitlab-operator.yaml>`__. + `Example of upgrade + <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-clusters-conf/-/commit/1a349433d45623128deabe7ca4ca57b740e16cba>`__ - Commit and push the change. - - Wait a couple of minutes to let ArgoCD applies the change - - A restart of the operator pod should happen (check the gitlab-controller-manager pod). + - Wait a couple of minutes to let ArgoCD apply the change + - A restart of the operator pod should happen (check the gitlab-controller-manager + pod). - Something like this should be visible in the operator logs: .. code:: shell @@ -89,17 +109,23 @@ It's important to follow the operatore releases to not be more than 3 minor vers > kubernetes logs gitlab-controller-manager-xxxx-xxxx Configuration error detected: chart version 6.5.1 not supported; please use one of the following: 6.5.2, 6.4.4, 6.3.5 -At this point, The gitlab operator is updated but the Gitlab upgrade itself is not yet started. +At this point, the gitlab operator is updated but the Gitlab upgrade itself is not yet +started. -- Step 2 `Upgrade the gitlab version <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/gitlab_upgrades.md>`__ +- Step 2 `Upgrade the gitlab version + <https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/master/doc/gitlab_upgrades.md>`__ - - Follow the recommended upgrade path depending of the current deployed version of the gitlab instance and the allowed versions of the operator - - Update the chart version matching the desired gitlab version (should be present on the release page of the operator) in the - `gitlab resource declaration <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-swh-private-data/-/blob/master/gitlab-staging/gitlab-staging.yaml>`__ + - Follow the recommended upgrade path depending on the current deployed version of + the gitlab instance and the allowed versions of the operator + - Update the chart version matching the desired gitlab version (should be present on + the release page of the operator) in the `gitlab resource declaration + <https://gitlab.softwareheritage.org/infra/ci-cd/k8s-swh-private-data/-/blob/master/gitlab-staging/gitlab-staging.yaml>`__ - Commit and push the change - - Wait a couple of minutes to let ArgoCD applies the change - - A upgrade of the gitlab services should happen and the launch of a migration pod should be done - - After a couple of minutes (could be as long as an hour), the cluster should stabilize if everything is ok (no pending pods, no pods in error, ...) + - Wait a couple of minutes to let ArgoCD apply the change + - An upgrade of the gitlab services should happen and the launch of a migration pod + should be done + - After a couple of minutes (could be as long as an hour), the cluster should + stabilize if everything is ok (no pending pods, no pods in error, ...) - Check the deployed version and the operator logs to check if nothing is wrong .. code:: shell