production/web: Align archive instance with previous instance moma

Review changes
Download
Patches
Plain diff

Merged production/web: Align archive instance with previous instance moma

align-web-configuration-with-moma into staging

Overview 25
Commits 2
Pipelines 0
Changes 1

Merged Antoine R. Dumont requested to merge align-web-configuration-with-moma into staging 1 year ago

Overview 25
Commits 2
Pipelines 0
Changes 1

Compared to moma, it's currently undersized to 2 replicas, 2 workers, 2 threads so 8 requests handling. As no gunicorn configuration is provided, the default values are set by the docker image's default values set in the Dockerfile.

In moma, it was configured with 32 workers (with a timeout of 3600s). So we bump the number of replicas to 4, and the number of workers to 4 as well. That makes up a total of 32 requests that can be handled, which matches what moma used to do.

This doubles the memory request since we doubled the number of workers, and current use is near 95% use of memory. This also aligns the request timeout to 3600s.

Another commit adapts the web template so it uses the same default as the Dockerfile declares. This has the benefit to explicit the current setup in environment variables (without impact with regards to the current setup)

make swh-helm-diff

[swh] Comparing changes between branches production and align-web-configuration-with-moma (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in align-web-configuration-with-moma branch for environment staging...
[swh] Generate config in align-web-configuration-with-moma branch for environment staging...
[swh] Generate config in align-web-configuration-with-moma branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in align-web-configuration-with-moma branch for environment production...
[swh] Generate config in align-web-configuration-with-moma branch for environment production...
[swh] Generate config in align-web-configuration-with-moma branch for environment production...


------------- diff for environment staging namespace swh -------------

--- /tmp/swh-chart.swh.eDzE0c3T/staging-swh.before      2024-01-16 18:07:44.478771192 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/staging-swh.after       2024-01-16 18:07:45.110766624 +0100
@@ -24621,20 +24621,26 @@
                   value: webapp-postgresql.internal.staging.swh.network
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging


------------- diff for environment staging namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra.before    2024-01-16 18:07:44.682769718 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra.after     2024-01-16 18:07:45.314765150 +0100
@@ -23105,20 +23105,26 @@
                   value: webapp.staging.swh.network
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging


------------- diff for environment staging namespace swh-cassandra-next-version -------------

--- /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra-next-version.before       2024-01-16 18:07:44.878768301 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra-next-version.after        2024-01-16 18:07:45.506763762 +0100
@@ -21282,20 +21282,26 @@
                   value: webapp-cassandra-next-version.internal.staging.swh.network
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging


------------- diff for environment production namespace swh -------------

--- /tmp/swh-chart.swh.eDzE0c3T/production-swh.before   2024-01-16 18:07:45.778761796 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/production-swh.after    2024-01-16 18:07:46.210758674 +0100
@@ -29902,20 +29902,26 @@
                   value: webapp1.internal.softwareheritage.org
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: production
@@ -29985,21 +29991,21 @@
 # Source: swh/templates/web/deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   namespace: swh
   name: web-archive
   labels:
     app: web-archive
 spec:
   revisionHistoryLimit: 2
-  replicas: 2
+  replicas: 4
   selector:
     matchLabels:
       app: web-archive
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
@@ -30115,21 +30121,21 @@
           args:
             - -c
             - cp -r $PWD/.local/share/swh/web/static/ /usr/share/swh/web/static/
           volumeMounts:
           - name: static
             mountPath: /usr/share/swh/web/static
       containers:
         - name: web-archive
           resources:
             requests:
-              memory: 3072Mi
+              memory: 6144Mi
               cpu: 350m
           image: container-registry.softwareheritage.org/swh/infra/swh-apps/web:20240111.1
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5004
               name: webapp
           readinessProbe:
             httpGet:
               path: /
               port: webapp
@@ -30149,20 +30155,26 @@
                   value: archive.softwareheritage.org
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "4"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: production


------------- diff for environment production namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.eDzE0c3T/production-swh-cassandra.before 2024-01-16 18:07:45.938760640 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/production-swh-cassandra.after  2024-01-16 18:07:46.366757547 +0100
@@ -14907,20 +14907,26 @@
                   value: webapp-cassandra.internal.softwareheritage.org
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: production

Refs. swh/infra/sysadm-environment#5110 (closed)

Edited 1 year ago by Antoine R. Dumont

Merge request reports

0 Assignees

None

Select assignees

0 Reviewers

Request review from

Labels

None

Select labels

Manage project labels

Milestone

None

Time tracking

No estimate or time spent

0 Participants

Viewing commit ab92184a

Show latest version

1 file

+ 4

− 6

Compare changes

Side-by-side
Inline

Show whitespace changes

Show one file at a time

Verified

ab92184a

template/web: Simplify the gunicorn setup · ab92184a

Antoine R. Dumont authored 1 year ago

By default, if nothing is provided, 2 workers, 2 threads and a request timeout of 3600s.
This matches the default installed in the Dockerfile for swh-web.

Refs. swh/infra/sysadm-environment#5110

swh/templates/web/deployment.yaml

+ 4

− 6

@@ -163,14 +163,12 @@ spec:
            - -c
            - /opt/swh/entrypoint.sh
          env:
            {{ if $web_config.gunicorn -}}
            - name: THREADS
              value: {{ $web_config.gunicorn.threads | default 5 | quote }}
            - name: WORKERS
              value: {{ $web_config.gunicorn.workers | default 2 | quote }}
              value: {{ dig "gunicorn" "workers" 2 $web_config | quote }}
            - name: THREADS
              value: {{ dig "gunicorn" "threads" 2 $web_config | quote }}
            - name: TIMEOUT
              value: {{ $web_config.gunicorn.timeout | default 60 | quote }}
            {{ end -}}
              value: {{ dig "gunicorn" "timeout" 3600 $web_config | quote }}
            - name: STATSD_HOST
              value: {{ $.Values.statsdExternalHost | default "prometheus-statsd-exporter" }}
            - name: STATSD_PORT