Bootstrap a gitlab instance hosted on our infrastructure

In order to prepare the gitlab migration, we should look at the different way to install and manage a gitlab instance on our own.

This page lists different installation methods: https://docs.gitlab.com/ee/install/

---

Migrated from T4046 (view on Phabricator)

added GitLab migration System administration priority:Normal state:wip labels

The following installation methods were tested:

1. debian packages
2. docker image
3. helm charts
4. gitlab operator

1/ and 2/ are working well but are installing a lot of re-packaged software like nginx, postgresql, redis, prometheus, grafana. It will force us to implement some login in our puppet code to manage the configuration as they are managed in the gitlab way. Zero-downtime upgrades are not possible with a single-node deployment and must manually managed if a multi-node deployment is configured

3/ The deployed component are finely tunable and only the activated components will be deployed in the cluster. The zero-downtime upgrade is not yet implemented with this kind of deployment [1] The upgrades are completely managed through the helm chart and are resumed to a one liner upgrade

4/ The gitlab operator is not yet production ready but is under active development. The viable version[2] is scheduled for may so should be available before our final migration. The gitlab operator is using the helm chart but manage all the life cycle of the deployed components. The biggest advantage is the upgrade are fully managed by the operator and are performed with zero downtime.

According to the tests with an AKS cluster, a minimal configuration with enough dynamic redundancy to allow the upgrades should cost around ~300

/m without the storage (probably Roadmap 2021

/m)

The next step is to test a deployment though terraform, and what kind of monitoring we could implement.

• [1] https://docs.gitlab.com/charts/installation/upgrade.html
• [2] https://about.gitlab.com/direction/maturity/

mentioned in commit swh-sysadmin-provisioning@3b97e0d5

The test instance can be reached at https://gitlab-staging.swh.network

This is a draft for the manual installation: https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w#

Several points need to be fixed like:

• The ability to send a mail through pergamon like the other servers
• A persistent ip address not changing after each ingress controller redeployment
• a decent monitoring

mentioned in commit swh-sysadmin-provisioning@c7097d2a

Status update:

• The global installation process is defined
  • The static ip is configured during the terraform deployment
• The ability to monitor the cluster and the gitlab is verified
  • Prometheus url with the kubernetes stats (for federation) : http://192.168.200.15:9090
  • Temporary grafana to have an idea of the possible dashboards: http://192.168.200.15:3000
  • Gitlab exported stats (exporter to add on the pergamon's prometheus):
    • Readiness probe: https://gitlab-staging.swh.network/-/readiness?token=
    • Liveness probe: https://gitlab-staging.swh.network/-/liveness?token=
    • Prometheus exporter: https://gitlab-staging.swh.network/-/metrics?token= The token can be found on the dedicated gitlab page: https://gitlab-staging.swh.network/admin/health_check
• outbound emails: Solved by creating a gandi's inbox and configuring the deployment to use it

All the configuration / deployment steps are listed on the hedgedoc document: https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w#

The next step is to test the backups and the restoration on our infrastructure

mentioned in commit swh-sysadmin-provisioning@b6f1b781

mentioned in commit swh-sysadmin-provisioning@2630acf0

mentioned in commit swh-sysadmin-provisioning@7edb55db

mentioned in commit swh-sysadmin-provisioning@0241d0e4

mentioned in commit swh-sysadmin-provisioning@5292d8c4

A restoration of the azure instance on our infra was successfully performed [1]. Everything is well imported: users, repositories, issues, ... The usage of a quick and dirty longhorn storage seems to make the instance slower than azure but the performance was not the goal of this POC.

I used this test to initialize a rancher instance allowing to manage our future internal kubernetes clusters [2]. #4144 (closed) will be used to test if it will be possible to manage the clusters and their nodes with terraform.

• [1] https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w?both#Migration-to-another-kubernetes-cluster
• [2] https://hedgedoc.softwareheritage.org/ynL9z-6JRnGYhW8vrLDl-w?both#Rancher-installation

removed state:wip label

closed

mentioned in issue #4063 (closed)