Backup the gitlab production instance
The production gitlab instance must be backuped outside azure to increase the resiliency
- azure blob storage must be backuped
- database and repositories must be backuped
- Limit the history of the backups stored on minio
Some resources:
Activity
added GitLab migration priority:High labels
marked this issue as related to #4063 (closed)
added environment: production label
mentioned in issue #4063 (closed)
marked this issue as related to #4663 (closed)
added state:wip label
The data of the gitlab staging backup is configured to run each day at 23:00 UTC.
The tar are stored here: https://minio-console.internal.admin.swh.network/buckets/backup-gitlab-staging/browseA dedicated script is now needed to backup the object storage content into minio
Isn't the GitLab backup tool supposed to back up the object storage contents as well?
https://docs.gitlab.com/charts/architecture/backup-restore.html#sequence-of-execution ?
ah, it looks like there is a difference according to where we look in the documentation. According to https://docs.gitlab.com/ee/raketasks/backup_restore.html it seems to be not backuped
GitLab doesn’t back up items that aren’t stored on the file system. If you’re using object storage, be sure to enable backups with your object storage provider, if desired.
I will check that
It seems the backup tool is not supporting azure object storage: https://gitlab.com/gitlab-org/build/CNG/-/blob/f65867afa54f6d0033e19f9e9038ec680abd5eb2/gitlab-toolbox/scripts/bin/backup-utility#L232-266I'm not very confident it will workEdit: The section is the copy of the backuped items to the remote object storage.
Edited by Vincent Sellier-
That's the "copy backups to object storage" part, not the "get tarball of stuff that needs to be backed up" part. What's relevant is this line:
This uses gitlab rake so it will support any object storage backend supported by gitlab, supposedly.
-
https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/1138 adds support for azure in the gitlab backup scripts. Unfortunately it looks like this will miss the next merge train...
For the record, gitlab staging was restored with the content of the production.
The steps were:- In minio, copy the backup from the
backup-gitlabbucket to thebackup-gitlab-stagingbucket (download + upload from the minio console) - Connect to the gitlab staging toolbox and launch:
# kubectl --context euwest-gitlab-staging exec -ti deployment/gitlab-toolbox bash git@gitlab-toolbox-7db8f69984-wm9dr:/$ backup-utility --restore -t 1669229222_2022_11_23_15.5.2 INFO: Retrieving list of remote files for s3://backup-gitlab-staging/1669229222_2022_11_23_15.5.2_gitlab_backup.tar ... INFO: Summary: 1 remote files to download INFO: Receiving file '/srv/gitlab/tmp/backups/0_gitlab_backup.tar', please wait... Unpacking backup 2022-11-23 23:01:05 +0000 -- Restoring main_database ... 2022-11-23 23:01:05 +0000 -- Be sure to stop Puma, Sidekiq, and any other process that connects to the database before proceeding. For Omnibus installs, see the following link for more information: https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations Before restoring the database, we will remove all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed. Do you want to continue (yes/no)? ... 2022-11-23 23:09:33 +0000 -- Restoring repositories ... done 2022-11-23 23:09:33 +0000 -- Deleting backup and restore lock file- replicate the production blob storage:
- copy locally the production content (registry + uploads because other buckets are empty)
# mkdir registry uploads # # Copy registry content (path is hacked because there is an empty directory in the path and azcopy don't want to sync it directly) azcopy sync "https://swheuwestgitlabprod.blob.core.windows.net/registry//docker?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED" registry # # Copy uploads content azcopy sync "https://swheuwestgitlabprod.blob.core.windows.net/uploads?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED" uploads # # Cleanup staging buckets azcopy remove "https://swheuwestgitlabstaging.blob.core.windows.net/registry//?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED" azcopy remove "https://swheuwestgitlabstaging.blob.core.windows.net/uploads/@hashed?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED" azcopy remove "https://swheuwestgitlabstaging.blob.core.windows.net/uploads/users?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED" # # Copy buckets to staging azcopy sync --recursive registry "https://swheuwestgitlabstaging.blob.core.windows.net/registry//docker/registry?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED" azcopy sync --recursive uploads "https://swheuwestgitlabstaging.blob.core.windows.net/uploads?sv=2021-06-08&ss=bfqt&srt=sco&sp=rltfx&se=2022-11-24T16:15:02Z&st=2022-11-24T08:15:02Z&spr=https&sig=REDACTED"The credentials can be created in the storage account, section "Network and Security" / "Shared Access Signature" or "Signature d'accès partagé" in french version, adapt the checkboxes according the needs and generate the signature. Use the generated BLOB url.
Edited by Vincent Sellier- In minio, copy the backup from the
Some secrets need to be restored too to have a fully working instance (2fa, ...):
kubectl --context euwest-gitlab-production -n gitlab-system get secrets gitlab-rails-secret -o yaml > gitlab-rails-secret-production.yaml kubectl --context euwest-gitlab-staging delete -n gitlab-system secret gitlab-rails-secret kubectl --context euwest-gitlab-staging apply -n gitlab-system -f gitlab-rails-secret-production.yaml kubectl rollout restart deployment gitlab-toolbox gitlab-webservice-default gitlab-sidekiq-all-in-1-v2 gitlab-registryThis secret must be also integrated in the backup (need to check if it could be integrated in a backup of the k8s cluster directly)
Edited by Vincent Sellier-
There's a bunch of secrets that can get generated by the gitlab charts. Some of them are used to encrypt data at rest. I think it would make sense to set most of them manually in our k8s cluster config (and make sure that the ones that are used for encryption at rest are synced between staging and prod, at least while we use staging out of prod backups)
Manual secrets creation seems to be well supported: https://docs.gitlab.com/charts/installation/secrets.html#manual-secret-creation-optional
The only thing that's not clear to me is how manual secret updates are mentioned in release notes. We should figure that out during the GitLab 15.6/Helm chart 6.6 upgrade, I guess.
-
Actually it seems that data encryption at rest is only handled through the rails-secret. Maybe? At least we should start keeping that in the k8s secrets repo.
All other secrets seem to be transient secrets for internal communication between GitLab components. Hard to guess from just the chart docs though.
changed milestone to %Finish GitLab migration
assigned to @vsellier
After digging, it seems the rclone tools can sync azure to s3 buckets.
It's packaged in debian and seems to work pretty well at the cost of a very little configuration.For example, to for a sync of the staging
uploadsbucket:rclone sync -v azure:uploads minio:backup-gitlab-staging/object-storage/uploads ... 2023/02/09 11:30:42 INFO : @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/ffe3c68167636869ab3ac25827eac7b6/deployment.png: Copied (new) 2023/02/09 11:30:42 INFO : @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/f2ea97f2fa7b2a927b5e02c07527638a/Statement_of_interest.pdf: Copied (new) 2023/02/09 11:30:42 INFO : Waiting for deletions to finish 2023/02/09 11:30:42 INFO : Transferred: 227.513M / 227.513 MBytes, 100%, 4.180 MBytes/s, ETA 0s Errors: 0 Checks: 445 / 445, 100% Transferred: 861 / 861, 100% Elapsed time: 54.4sThe only needed configuration:
root@2e73b5b1f1a3:~/.config/rclone# cat ~/.config/rclone/rclone.conf [azure] type = azureblob account = <redacted> sas_url = https://<redacted>.blob.core.windows.net/?sv=<redacted> [minio] type = s3 provider = Minio env_auth = false access_key_id = gitlab<redacted> secret_access_key = <redacted> acl = private endpoint = https://minio.<redacted>We should be able to configure kubernetes jobs to regularly perform the synchronization.
More jobs will be needed to:
- create an archive of the buckets to have snapshots and
- limit the history of the archive in the bucket
Edited by Vincent Sellier
