Prepare deposit template & deploy to staging for a first round
The first commit prepares the deposit template to deploy instance in clusters. The template provide an extra django settings file so we can configure extra backends to be used with an azure storage (for now).
azure was chosen because we already have the necessary tools to configure this simply. Using s3 would have involved more cogs (starting by deploying a minio or some such and we don't have any chart/template for that yet). This can always be revisited later.
The second commit actually deploys a deposit instance in the staging cluster.
Tested through minikube.
The only part which is not tested exhaustively [1] is the wiring to the actual azure storage through the extra django settings file. I'd like to deploy this and test it on staging [2].
[1] fixed by fixing up until it no longer crashed at startup time within minikube (hence the double "" within the container name and connection string too). That might change with real life testing.
[2] I don't really see another simple way of testing this. We are missing too many cogs (hard work to wire too) within minikube to test it thoroughly.
helm diff
[swh] Comparing changes between branches production and staging-prepare-deposit-template (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in staging-prepare-deposit-template branch for environment staging...
[swh] Generate config in staging-prepare-deposit-template branch for environment staging...
[swh] Generate config in staging-prepare-deposit-template branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in staging-prepare-deposit-template branch for environment production...
[swh] Generate config in staging-prepare-deposit-template branch for environment production...
[swh] Generate config in staging-prepare-deposit-template branch for environment production...
------------- diff for environment staging namespace swh -------------
--- /tmp/swh-chart.swh.uqAmYt3i/staging-swh.before 2023-11-10 15:03:23.059837626 +0100
+++ /tmp/swh-chart.swh.uqAmYt3i/staging-swh.after 2023-11-10 15:03:23.711836266 +0100
@@ -14035,20 +14035,30 @@
port: 443
protocol: TCP
targetPort: 9443
selector:
app: keda-admission-webhooks
---
# Source: swh/templates/external-services/cname.yaml
apiVersion: v1
kind: Service
metadata:
+ name: deposit-rpc-ingress
+ namespace: swh
+spec:
+ type: ExternalName
+ externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
name: indexer-storage-rpc-ingress
namespace: swh
spec:
type: ExternalName
externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
---
# Source: swh/templates/external-services/cname.yaml
apiVersion: v1
kind: Service
metadata:
------------- diff for environment staging namespace swh-cassandra -------------
--- /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra.before 2023-11-10 15:03:23.267837192 +0100
+++ /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra.after 2023-11-10 15:03:23.915835840 +0100
@@ -334,20 +334,77 @@
swh:
level: "INFO"
celery.task:
level: "INFO"
root:
level: "INFO"
handlers:
- console
---
+# Source: swh/templates/deposit/configmap.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ namespace: swh-cassandra
+ name: deposit-configuration-template
+data:
+ config.yml.template: |
+ instance_name: deposit.staging.swh.network
+ allowed_hosts:
+ - deposit.staging.swh.network
+ swh_authority_url: deposit.staging.swh.network
+ storage:
+ cls: remote
+ url: http://storage:5002
+ storage_metadata:
+ cls: remote
+ url: http://storage:5002
+ scheduler:
+ cls: remote
+ url: http://scheduler.internal.staging.swh.network
+ authentication_provider: keycloak
+ keycloak:
+ realm_name: SoftwareHeritageStaging
+ server_url: https://auth.softwareheritage.org/auth/
+
+ private:
+ secret_key: ${DJANGO_SECRET_KEY}
+ db:
+
+ host: db1.internal.staging.swh.network
+ port: 5432
+ name: swh-deposit
+ user: swh-deposit
+ password: ${POSTGRESQL_PASSWORD}
+ cache_uri: memcached:11211
+ extraction_dir: /tmp/swh-deposit/archive/
+ max_upload_size: 209715200
+
+ deposit_settings.yaml.template: |
+ BACKEND: "storages.backends.azure_storage.AzureStorage"
+ OPTIONS:
+ azure_container: ""deposit-contents""
+ connection_string: ""DefaultEndpointsProtocol=https;AccountName=swhdepositstoragestaging;AccountKey=${ACCOUNT_KEY};EndpointSuffix=core.windows.net""
+
+ deposit_settings.py: |
+ from swh.deposit.settings.production import * # noqa
+
+ from pathlib import Path
+ from yaml import safe_load
+ backend = safe_load(Path('/etc/swh/deposit_settings.yaml').read_bytes())
+
+ STORAGES = {
+ "staticfiles": backend,
+ "default": backend,
+ }
+---
# Source: swh/templates/graphql/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: graphql
namespace: swh-cassandra
data:
# TODO: rename to not have a dot in the name to allow testing
config.yml: |
storage:
@@ -13662,20 +13719,45 @@
namespace: default
spec:
ports:
- name: http
port: 443
protocol: TCP
targetPort: 9443
selector:
app: keda-admission-webhooks
---
+# Source: swh/templates/deposit/service.yaml
+apiVersion: v1
+kind: Service
+metadata:
+ name: deposit
+ namespace: swh-cassandra
+spec:
+ type: ClusterIP
+ selector:
+ app: deposit
+ ports:
+ - port: 5006
+ targetPort: 5006
+ name: rpc
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
+ name: deposit-rpc-ingress
+ namespace: swh-cassandra
+spec:
+ type: ExternalName
+ externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
# Source: swh/templates/external-services/cname.yaml
apiVersion: v1
kind: Service
metadata:
name: indexer-storage-rpc-ingress
namespace: swh-cassandra
spec:
type: ExternalName
externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
---
@@ -14419,20 +14501,223 @@
defaultMode: 0777
items:
- key: "pre-stop-idempotent.sh"
path: "pre-stop.sh"
- name: sentry-settings-for-celery-tasks
secret:
secretName: sentry-settings-for-celery-tasks
optional: true
---
+# Source: swh/templates/deposit/deployment.yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ namespace: swh-cassandra
+ name: deposit
+ labels:
+ app: deposit
+spec:
+ revisionHistoryLimit: 2
+ selector:
+ matchLabels:
+ app: deposit
+ strategy:
+ type: RollingUpdate
+ rollingUpdate:
+ maxSurge: 1
+ template:
+ metadata:
+ labels:
+ app: deposit
+ annotations:
+ checksum/config: 32c1c61d7f9df1fdff944970d82d07d2a5dc68c09ec66f871d7a4c14adc2a609
+ spec:
+ affinity:
+ nodeAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ nodeSelectorTerms:
+ - matchExpressions:
+ - key: swh/deposit
+ operator: In
+ values:
+ - "true"
+ priorityClassName: swh-cassandra-frontend-rpc
+
+ initContainers:
+ - name: prepare-settings-with-secret
+ image: debian:bullseye
+ imagePullPolicy: IfNotPresent
+ command:
+ - /bin/bash
+ args:
+ - -c
+ - eval echo "\"$(</etc/swh/configuration-template/deposit_settings.yaml.template)\"" > /etc/swh/deposit_settings.yaml
+ env:
+
+
+ - name: ACCOUNT_KEY
+ valueFrom:
+ secretKeyRef:
+ name: swh-deposit-azure-secret
+ key: azure-swh-deposit-key
+ # 'name' secret must exist & include that ^ key
+ optional: false
+ volumeMounts:
+ - name: configuration
+ mountPath: /etc/swh
+ - name: configuration-template
+ mountPath: /etc/swh/configuration-template
+ - name: prepare-settings
+ image: debian:bullseye
+ imagePullPolicy: IfNotPresent
+ command:
+ - /bin/bash
+ args:
+ - -c
+ - cp /etc/swh/configuration-template/deposit_settings.py /etc/swh/deposit_settings.py
+ volumeMounts:
+ - name: configuration
+ mountPath: /etc/swh
+ - name: configuration-template
+ mountPath: /etc/swh/configuration-template
+ - name: prepare-configuration
+ image: debian:bullseye
+ imagePullPolicy: IfNotPresent
+ command:
+ - /bin/bash
+ args:
+ - -c
+ - eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml
+ env:
+
+ - name: POSTGRESQL_PASSWORD
+ valueFrom:
+ secretKeyRef:
+ name: swh-postgresql-deposit-secrets
+ key: postgres-swh-deposit-password
+ # 'name' secret must exist & include that ^ key
+ optional: false
+
+ - name: DJANGO_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: swh-deposit-django-secret
+ key: deposit-django-secret-key
+ # 'name' secret must exist & include that ^ key
+ optional: false
+
+ - name: SWH_SENTRY_DSN
+ valueFrom:
+ secretKeyRef:
+ name: common-secrets
+ key: deposit-sentry-dsn
+ optional: false
+
+ volumeMounts:
+ - name: configuration
+ mountPath: /etc/swh
+ - name: configuration-template
+ mountPath: /etc/swh/configuration-template
+ containers:
+ - name: deposit
+ resources:
+ requests:
+ memory: 500Mi
+ cpu: 500m
+ image: container-registry.softwareheritage.org/swh/infra/swh-apps/deposit:20231109.1
+ imagePullPolicy: IfNotPresent
+ ports:
+ - containerPort: 5006
+ name: deposit-app
+ readinessProbe:
+ httpGet:
+ path: /
+ port: deposit-app
+ httpHeaders:
+ - name: Host
+ value: deposit.staging.swh.network
+ initialDelaySeconds: 5
+ failureThreshold: 30
+ periodSeconds: 10
+ timeoutSeconds: 30
+ livenessProbe:
+ httpGet:
+ path: /
+ port: deposit-app
+ httpHeaders:
+ - name: Host
+ value: deposit.staging.swh.network
+ initialDelaySeconds: 3
+ periodSeconds: 10
+ timeoutSeconds: 30
+ command:
+ - /bin/bash
+ args:
+ - -c
+ - /opt/swh/entrypoint.sh
+ env:
+ - name: STATSD_HOST
+ value: prometheus-statsd-exporter
+ - name: STATSD_PORT
+ value: "9125"
+ - name: LOG_LEVEL
+ value: "INFO"
+ - name: SWH_CONFIG_FILENAME
+ value: /etc/swh/config.yml
+ - name: DJANGO_SETTINGS_MODULE
+ value: deposit_settings
+ - name: PYTHONPATH
+ value: /etc/swh
+ - name: SWH_SENTRY_ENVIRONMENT
+ value: staging
+ - name: SWH_MAIN_PACKAGE
+ value: swh.deposit
+ - name: SWH_SENTRY_DSN
+ valueFrom:
+ secretKeyRef:
+ name: common-secrets
+ key: deposit-sentry-dsn
+ optional: false
+ - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
+ value: "true"
+
+ - name: DJANGO_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ name: swh-deposit-django-secret
+ key: deposit-django-secret-key
+ # 'name' secret must exist & include that ^ key
+ optional: false
+
+ volumeMounts:
+ - name: configuration
+ mountPath: /etc/swh
+ readOnly: true
+ - name: localstorage
+ mountPath: /tmp
+ volumes:
+ - name: localstorage
+ emptyDir: {}
+ - name: configuration
+ emptyDir: {}
+ - name: configuration-template
+ configMap:
+ name: deposit-configuration-template
+ items:
+ - key: "deposit_settings.yaml.template"
+ path: "deposit_settings.yaml.template"
+ - key: "deposit_settings.py"
+ path: "deposit_settings.py"
+ - key: "config.yml.template"
+ path: "config.yml.template"
+---
# Source: swh/templates/graphql/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: graphql
namespace: swh-cassandra
labels:
app: graphql
spec:
revisionHistoryLimit: 2
@@ -21065,20 +21350,43 @@
emptyDir: {}
- name: configuration-template
configMap:
name: web-configuration-template
items:
- key: "config.yml.template"
path: "config.yml.template"
- name: static
emptyDir: {}
---
+# Source: swh/templates/deposit/autoscaling.yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+ namespace: swh-cassandra
+ name: deposit
+ labels:
+ app: deposit
+spec:
+ scaleTargetRef:
+ apiVersion: apps/v1
+ kind: Deployment
+ name: deposit
+ minReplicas: 2
+ maxReplicas: 3
+ metrics:
+ - type: Resource
+ resource:
+ name: cpu
+ target:
+ type: Utilization
+ averageUtilization: 100
+---
# Source: swh/templates/storage/autoscaling.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
namespace: swh-cassandra
name: storage
labels:
app: storage
spec:
scaleTargetRef:
@@ -21134,20 +21442,52 @@
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
---
+# Source: swh/templates/deposit/ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ namespace: swh-cassandra
+ name: deposit-ingress-default
+ annotations:
+
+ cert-manager.io/cluster-issuer: letsencrypt-production-gandi
+ kubernetes.io/ingress.class: nginx
+ kubernetes.io/tls-acme: "true"
+ nginx.ingress.kubernetes.io/ssl-redirect: "false"
+
+
+spec:
+ rules:
+ - host: deposit.staging.swh.network
+ http:
+ paths:
+ - path: /
+ pathType: Prefix
+ backend:
+ service:
+ name: deposit
+ port:
+ number: 5006
+
+ tls:
+ - hosts:
+ - deposit.staging.swh.network
+ secretName: swh-deposit-crt
+---
# Source: swh/templates/graphql/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: swh-cassandra
name: graphql-ingress-default
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 10.42.0.0/16,10.43.0.0/16,192.168.101.0/24,192.168.130.0/24,192.168.50.0/24
nginx.ingress.kubernetes.io/rewrite-target: /
------------- diff for environment staging namespace swh-cassandra-next-version -------------
--- /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra-next-version.before 2023-11-10 15:03:23.467836775 +0100
+++ /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra-next-version.after 2023-11-10 15:03:24.123835407 +0100
@@ -13460,20 +13460,30 @@
port: 443
protocol: TCP
targetPort: 9443
selector:
app: keda-admission-webhooks
---
# Source: swh/templates/external-services/cname.yaml
apiVersion: v1
kind: Service
metadata:
+ name: deposit-rpc-ingress
+ namespace: swh-cassandra-next-version
+spec:
+ type: ExternalName
+ externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
name: indexer-storage-rpc-ingress
namespace: swh-cassandra-next-version
spec:
type: ExternalName
externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
---
# Source: swh/templates/external-services/cname.yaml
apiVersion: v1
kind: Service
metadata:
------------- diff for environment production namespace swh -------------
No differences
------------- diff for environment production namespace swh-cassandra -------------
No differences