Prepare deposit template & deploy to staging for a first round

The first commit prepares the deposit template to deploy instance in clusters. The template provide an extra django settings file so we can configure extra backends to be used with an azure storage (for now).

azure was chosen because we already have the necessary tools to configure this simply. Using s3 would have involved more cogs (starting by deploying a minio or some such and we don't have any chart/template for that yet). This can always be revisited later.

The second commit actually deploys a deposit instance in the staging cluster.

Tested through minikube.

The only part which is not tested exhaustively [1] is the wiring to the actual azure storage through the extra django settings file. I'd like to deploy this and test it on staging [2].

[1] fixed by fixing up until it no longer crashed at startup time within minikube (hence the double "" within the container name and connection string too). That might change with real life testing.

[2] I don't really see another simple way of testing this. We are missing too many cogs (hard work to wire too) within minikube to test it thoroughly.

helm diff

[swh] Comparing changes between branches production and staging-prepare-deposit-template (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in staging-prepare-deposit-template branch for environment staging...
[swh] Generate config in staging-prepare-deposit-template branch for environment staging...
[swh] Generate config in staging-prepare-deposit-template branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in staging-prepare-deposit-template branch for environment production...
[swh] Generate config in staging-prepare-deposit-template branch for environment production...
[swh] Generate config in staging-prepare-deposit-template branch for environment production...


------------- diff for environment staging namespace swh -------------

--- /tmp/swh-chart.swh.uqAmYt3i/staging-swh.before      2023-11-10 15:03:23.059837626 +0100
+++ /tmp/swh-chart.swh.uqAmYt3i/staging-swh.after       2023-11-10 15:03:23.711836266 +0100
@@ -14035,20 +14035,30 @@
     port: 443
     protocol: TCP
     targetPort: 9443
   selector:
     app: keda-admission-webhooks
 ---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:
+  name: deposit-rpc-ingress
+  namespace: swh
+spec:
+  type: ExternalName
+  externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
   name: indexer-storage-rpc-ingress
   namespace: swh
 spec:
   type: ExternalName
   externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:


------------- diff for environment staging namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra.before    2023-11-10 15:03:23.267837192 +0100
+++ /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra.after     2023-11-10 15:03:23.915835840 +0100
@@ -334,20 +334,77 @@
       swh:
         level: "INFO"
       celery.task:
         level: "INFO"

     root:
       level: "INFO"
       handlers:
       - console
 ---
+# Source: swh/templates/deposit/configmap.yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  namespace: swh-cassandra
+  name: deposit-configuration-template
+data:
+  config.yml.template: |
+    instance_name: deposit.staging.swh.network
+    allowed_hosts:
+      - deposit.staging.swh.network
+    swh_authority_url: deposit.staging.swh.network
+    storage:
+      cls: remote
+      url: http://storage:5002
+    storage_metadata:
+      cls: remote
+      url: http://storage:5002
+    scheduler:
+      cls: remote
+      url: http://scheduler.internal.staging.swh.network
+    authentication_provider: keycloak
+    keycloak:
+      realm_name: SoftwareHeritageStaging
+      server_url: https://auth.softwareheritage.org/auth/
+
+    private:
+      secret_key: ${DJANGO_SECRET_KEY}
+      db:
+
+          host: db1.internal.staging.swh.network
+          port: 5432
+          name: swh-deposit
+          user: swh-deposit
+          password: ${POSTGRESQL_PASSWORD}
+    cache_uri: memcached:11211
+    extraction_dir: /tmp/swh-deposit/archive/
+    max_upload_size: 209715200
+
+  deposit_settings.yaml.template: |
+    BACKEND: "storages.backends.azure_storage.AzureStorage"
+    OPTIONS:
+      azure_container: ""deposit-contents""
+      connection_string: ""DefaultEndpointsProtocol=https;AccountName=swhdepositstoragestaging;AccountKey=${ACCOUNT_KEY};EndpointSuffix=core.windows.net""
+
+  deposit_settings.py: |
+    from swh.deposit.settings.production import *  # noqa
+
+    from pathlib import Path
+    from yaml import safe_load
+    backend = safe_load(Path('/etc/swh/deposit_settings.yaml').read_bytes())
+
+    STORAGES = {
+      "staticfiles": backend,
+      "default": backend,
+    }
+---
 # Source: swh/templates/graphql/configmap.yaml
 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: graphql
   namespace: swh-cassandra
 data:
   # TODO: rename to not have a dot in the name to allow testing
   config.yml: |
     storage:
@@ -13662,20 +13719,45 @@
   namespace: default
 spec:
   ports:
   - name: http
     port: 443
     protocol: TCP
     targetPort: 9443
   selector:
     app: keda-admission-webhooks
 ---
+# Source: swh/templates/deposit/service.yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: deposit
+  namespace: swh-cassandra
+spec:
+  type: ClusterIP
+  selector:
+    app: deposit
+  ports:
+    - port: 5006
+      targetPort: 5006
+      name: rpc
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: deposit-rpc-ingress
+  namespace: swh-cassandra
+spec:
+  type: ExternalName
+  externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:
   name: indexer-storage-rpc-ingress
   namespace: swh-cassandra
 spec:
   type: ExternalName
   externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ---
@@ -14419,20 +14501,223 @@
           defaultMode: 0777
           items:
           - key: "pre-stop-idempotent.sh"
             path: "pre-stop.sh"

       - name: sentry-settings-for-celery-tasks
         secret:
           secretName: sentry-settings-for-celery-tasks
           optional: true
 ---
+# Source: swh/templates/deposit/deployment.yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  namespace: swh-cassandra
+  name: deposit
+  labels:
+    app: deposit
+spec:
+  revisionHistoryLimit: 2
+  selector:
+    matchLabels:
+      app: deposit
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+  template:
+    metadata:
+      labels:
+        app: deposit
+      annotations:
+        checksum/config: 32c1c61d7f9df1fdff944970d82d07d2a5dc68c09ec66f871d7a4c14adc2a609
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: swh/deposit
+                operator: In
+                values:
+                - "true"
+      priorityClassName: swh-cassandra-frontend-rpc
+
+      initContainers:
+        - name: prepare-settings-with-secret
+          image: debian:bullseye
+          imagePullPolicy: IfNotPresent
+          command:
+            - /bin/bash
+          args:
+            - -c
+            - eval echo "\"$(</etc/swh/configuration-template/deposit_settings.yaml.template)\"" > /etc/swh/deposit_settings.yaml
+          env:
+
+
+            - name: ACCOUNT_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: swh-deposit-azure-secret
+                  key: azure-swh-deposit-key
+                  # 'name' secret must exist & include that ^ key
+                  optional: false
+          volumeMounts:
+            - name: configuration
+              mountPath: /etc/swh
+            - name: configuration-template
+              mountPath: /etc/swh/configuration-template
+        - name: prepare-settings
+          image: debian:bullseye
+          imagePullPolicy: IfNotPresent
+          command:
+            - /bin/bash
+          args:
+            - -c
+            - cp /etc/swh/configuration-template/deposit_settings.py /etc/swh/deposit_settings.py
+          volumeMounts:
+            - name: configuration
+              mountPath: /etc/swh
+            - name: configuration-template
+              mountPath: /etc/swh/configuration-template
+        - name: prepare-configuration
+          image: debian:bullseye
+          imagePullPolicy: IfNotPresent
+          command:
+            - /bin/bash
+          args:
+            - -c
+            - eval echo "\"$(</etc/swh/configuration-template/config.yml.template)\"" > /etc/swh/config.yml
+          env:
+
+            - name: POSTGRESQL_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: swh-postgresql-deposit-secrets
+                  key: postgres-swh-deposit-password
+                  # 'name' secret must exist & include that ^ key
+                  optional: false
+
+            - name: DJANGO_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: swh-deposit-django-secret
+                  key: deposit-django-secret-key
+                  # 'name' secret must exist & include that ^ key
+                  optional: false
+
+            - name: SWH_SENTRY_DSN
+              valueFrom:
+                secretKeyRef:
+                  name: common-secrets
+                  key: deposit-sentry-dsn
+                  optional: false
+
+          volumeMounts:
+            - name: configuration
+              mountPath: /etc/swh
+            - name: configuration-template
+              mountPath: /etc/swh/configuration-template
+      containers:
+        - name: deposit
+          resources:
+            requests:
+              memory: 500Mi
+              cpu: 500m
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/deposit:20231109.1
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 5006
+              name: deposit-app
+          readinessProbe:
+            httpGet:
+              path: /
+              port: deposit-app
+              httpHeaders:
+                - name: Host
+                  value: deposit.staging.swh.network
+            initialDelaySeconds: 5
+            failureThreshold: 30
+            periodSeconds: 10
+            timeoutSeconds: 30
+          livenessProbe:
+            httpGet:
+              path: /
+              port: deposit-app
+              httpHeaders:
+                - name: Host
+                  value: deposit.staging.swh.network
+            initialDelaySeconds: 3
+            periodSeconds: 10
+            timeoutSeconds: 30
+          command:
+            - /bin/bash
+          args:
+            - -c
+            - /opt/swh/entrypoint.sh
+          env:
+            - name: STATSD_HOST
+              value: prometheus-statsd-exporter
+            - name: STATSD_PORT
+              value: "9125"
+            - name: LOG_LEVEL
+              value: "INFO"
+            - name: SWH_CONFIG_FILENAME
+              value: /etc/swh/config.yml
+            - name: DJANGO_SETTINGS_MODULE
+              value: deposit_settings
+            - name: PYTHONPATH
+              value: /etc/swh
+            - name: SWH_SENTRY_ENVIRONMENT
+              value: staging
+            - name: SWH_MAIN_PACKAGE
+              value: swh.deposit
+            - name: SWH_SENTRY_DSN
+              valueFrom:
+                secretKeyRef:
+                  name: common-secrets
+                  key: deposit-sentry-dsn
+                  optional: false
+            - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
+              value: "true"
+
+            - name: DJANGO_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: swh-deposit-django-secret
+                  key: deposit-django-secret-key
+                  # 'name' secret must exist & include that ^ key
+                  optional: false
+
+          volumeMounts:
+          - name: configuration
+            mountPath: /etc/swh
+            readOnly: true
+          - name: localstorage
+            mountPath: /tmp
+      volumes:
+      - name: localstorage
+        emptyDir: {}
+      - name: configuration
+        emptyDir: {}
+      - name: configuration-template
+        configMap:
+         name: deposit-configuration-template
+         items:
+         - key: "deposit_settings.yaml.template"
+           path: "deposit_settings.yaml.template"
+         - key: "deposit_settings.py"
+           path: "deposit_settings.py"
+         - key: "config.yml.template"
+           path: "config.yml.template"
+---
 # Source: swh/templates/graphql/deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: graphql
   namespace: swh-cassandra
   labels:
     app: graphql
 spec:
   revisionHistoryLimit: 2
@@ -21065,20 +21350,43 @@
         emptyDir: {}
       - name: configuration-template
         configMap:
          name: web-configuration-template
          items:
          - key: "config.yml.template"
            path: "config.yml.template"
       - name: static
         emptyDir: {}
 ---
+# Source: swh/templates/deposit/autoscaling.yaml
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  namespace: swh-cassandra
+  name: deposit
+  labels:
+    app: deposit
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: deposit
+  minReplicas: 2
+  maxReplicas: 3
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 100
+---
 # Source: swh/templates/storage/autoscaling.yaml
 apiVersion: autoscaling/v2
 kind: HorizontalPodAutoscaler
 metadata:
   namespace: swh-cassandra
   name: storage
   labels:
     app: storage
 spec:
   scaleTargetRef:
@@ -21134,20 +21442,52 @@
   minReplicas: 1
   maxReplicas: 3
   metrics:
   - type: Resource
     resource:
       name: cpu
       target:
         type: Utilization
         averageUtilization: 50
 ---
+# Source: swh/templates/deposit/ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  namespace: swh-cassandra
+  name: deposit-ingress-default
+  annotations:
+
+    cert-manager.io/cluster-issuer: letsencrypt-production-gandi
+    kubernetes.io/ingress.class: nginx
+    kubernetes.io/tls-acme: "true"
+    nginx.ingress.kubernetes.io/ssl-redirect: "false"
+
+
+spec:
+  rules:
+  - host: deposit.staging.swh.network
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: deposit
+            port:
+              number: 5006
+
+  tls:
+  - hosts:
+    - deposit.staging.swh.network
+    secretName: swh-deposit-crt
+---
 # Source: swh/templates/graphql/ingress.yaml
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   namespace: swh-cassandra
   name: graphql-ingress-default
   annotations:
     nginx.ingress.kubernetes.io/whitelist-source-range: 10.42.0.0/16,10.43.0.0/16,192.168.101.0/24,192.168.130.0/24,192.168.50.0/24

     nginx.ingress.kubernetes.io/rewrite-target: /


------------- diff for environment staging namespace swh-cassandra-next-version -------------

--- /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra-next-version.before       2023-11-10 15:03:23.467836775 +0100
+++ /tmp/swh-chart.swh.uqAmYt3i/staging-swh-cassandra-next-version.after        2023-11-10 15:03:24.123835407 +0100
@@ -13460,20 +13460,30 @@
     port: 443
     protocol: TCP
     targetPort: 9443
   selector:
     app: keda-admission-webhooks
 ---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:
+  name: deposit-rpc-ingress
+  namespace: swh-cassandra-next-version
+spec:
+  type: ExternalName
+  externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
   name: indexer-storage-rpc-ingress
   namespace: swh-cassandra-next-version
 spec:
   type: ExternalName
   externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:


------------- diff for environment production namespace swh -------------

No differences


------------- diff for environment production namespace swh-cassandra -------------

No differences

Refs. swh/infra/sysadm-environment#4780 (closed)

Refs. swh/infra/sysadm-environment#5129 (closed)

cluster-components/templates/alerting/cassandra-alerting.yaml

@@ -6,13 +6,13 @@ kind: PrometheusRule
 metadata:
   labels:
     app: cassandra
-  name: critical-cassandra-service.rules
+  name: cassandra-service.rules
   namespace: cattle-monitoring-system
 spec:
   groups:
-  - name: critical-cassandra-service.rules
+  - name: cassandra-service.rules
     rules:
-    - alert: Cassandra_Service_Degraded_In_{{ .environment | title }}
+    - alert: Cassandra_Degraded_Service_In_{{ .environment | title }}
       annotations:
         description: "The {{"{{"}} $labels.instance {{"}}"}} node is unreachable for more than {{ trimSuffix "m" .period.down }} minutes. This node seems down."
         summary: "The {{"{{"}} $labels.service {{"}}"}} is degraded. Please check the {{"{{"}} $labels.instance {{"}}"}} status."
@@ -21,7 +21,7 @@ spec:
       labels:
         severity: warning
         namespace: cattle-monitoring-system
-    - alert: Cassandra_Table_Unrepaired_In_{{ .environment | title }}
+    - alert: Cassandra_Unrepaired_Table_In_{{ .environment | title }}
       annotations:
         description: "The unrepaired bytes of table {{"{{"}} $labels.table {{"}}"}} is more than {{ div .unrepairedSize 1073741824 }} Gb."
         summary: "Please trigger a repair on the table {{"{{"}} $labels.table {{"}}"}} in keyspace {{"{{"}} $labels.keyspace {{"}}"}}."