production/web: Align archive instance with previous instance moma

Review changes
Download
Patches
Plain diff

Merged production/web: Align archive instance with previous instance moma

align-web-configuration-with-moma into staging

Overview 25
Commits 2
Pipelines 0
Changes 1

All threads resolved!

Merged Antoine R. Dumont requested to merge align-web-configuration-with-moma into staging 1 year ago

Overview 25
Commits 2
Pipelines 0
Changes 1

All threads resolved!

Compared to moma, it's currently undersized to 2 replicas, 2 workers, 2 threads so 8 requests handling. As no gunicorn configuration is provided, the default values are set by the docker image's default values set in the Dockerfile.

In moma, it was configured with 32 workers (with a timeout of 3600s). So we bump the number of replicas to 4, and the number of workers to 4 as well. That makes up a total of 32 requests that can be handled, which matches what moma used to do.

This doubles the memory request since we doubled the number of workers, and current use is near 95% use of memory. This also aligns the request timeout to 3600s.

Another commit adapts the web template so it uses the same default as the Dockerfile declares. This has the benefit to explicit the current setup in environment variables (without impact with regards to the current setup)

make swh-helm-diff

[swh] Comparing changes between branches production and align-web-configuration-with-moma (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in align-web-configuration-with-moma branch for environment staging...
[swh] Generate config in align-web-configuration-with-moma branch for environment staging...
[swh] Generate config in align-web-configuration-with-moma branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in align-web-configuration-with-moma branch for environment production...
[swh] Generate config in align-web-configuration-with-moma branch for environment production...
[swh] Generate config in align-web-configuration-with-moma branch for environment production...


------------- diff for environment staging namespace swh -------------

--- /tmp/swh-chart.swh.eDzE0c3T/staging-swh.before      2024-01-16 18:07:44.478771192 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/staging-swh.after       2024-01-16 18:07:45.110766624 +0100
@@ -24621,20 +24621,26 @@
                   value: webapp-postgresql.internal.staging.swh.network
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging


------------- diff for environment staging namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra.before    2024-01-16 18:07:44.682769718 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra.after     2024-01-16 18:07:45.314765150 +0100
@@ -23105,20 +23105,26 @@
                   value: webapp.staging.swh.network
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging


------------- diff for environment staging namespace swh-cassandra-next-version -------------

--- /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra-next-version.before       2024-01-16 18:07:44.878768301 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/staging-swh-cassandra-next-version.after        2024-01-16 18:07:45.506763762 +0100
@@ -21282,20 +21282,26 @@
                   value: webapp-cassandra-next-version.internal.staging.swh.network
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging


------------- diff for environment production namespace swh -------------

--- /tmp/swh-chart.swh.eDzE0c3T/production-swh.before   2024-01-16 18:07:45.778761796 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/production-swh.after    2024-01-16 18:07:46.210758674 +0100
@@ -29902,20 +29902,26 @@
                   value: webapp1.internal.softwareheritage.org
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: production
@@ -29985,21 +29991,21 @@
 # Source: swh/templates/web/deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   namespace: swh
   name: web-archive
   labels:
     app: web-archive
 spec:
   revisionHistoryLimit: 2
-  replicas: 2
+  replicas: 4
   selector:
     matchLabels:
       app: web-archive
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
@@ -30115,21 +30121,21 @@
           args:
             - -c
             - cp -r $PWD/.local/share/swh/web/static/ /usr/share/swh/web/static/
           volumeMounts:
           - name: static
             mountPath: /usr/share/swh/web/static
       containers:
         - name: web-archive
           resources:
             requests:
-              memory: 3072Mi
+              memory: 6144Mi
               cpu: 350m
           image: container-registry.softwareheritage.org/swh/infra/swh-apps/web:20240111.1
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5004
               name: webapp
           readinessProbe:
             httpGet:
               path: /
               port: webapp
@@ -30149,20 +30155,26 @@
                   value: archive.softwareheritage.org
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "4"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: production


------------- diff for environment production namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.eDzE0c3T/production-swh-cassandra.before 2024-01-16 18:07:45.938760640 +0100
+++ /tmp/swh-chart.swh.eDzE0c3T/production-swh-cassandra.after  2024-01-16 18:07:46.366757547 +0100
@@ -14907,20 +14907,26 @@
                   value: webapp-cassandra.internal.softwareheritage.org
             initialDelaySeconds: 3
             periodSeconds: 10
             timeoutSeconds: 30
           command:
             - /bin/bash
           args:
             - -c
             - /opt/swh/entrypoint.sh
           env:
+            - name: WORKERS
+              value: "2"
+            - name: THREADS
+              value: "2"
+            - name: TIMEOUT
+              value: "3600"
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: LOG_LEVEL
               value: "INFO"
             - name: SWH_CONFIG_FILENAME
               value: /etc/swh/config.yml
             - name: SWH_SENTRY_ENVIRONMENT
               value: production

Refs. swh/infra/sysadm-environment#5110 (closed)

Edited 1 year ago by Antoine R. Dumont

Merge request reports

0 Assignees

None

Select assignees

0 Reviewers

None

Request review from

Labels

GitLab migration

Select labels

Manage project labels

Milestone

None

Time tracking

No estimate or time spent

0 Participants

Viewing commit b0184d80

Show latest version

1 file

+ 4

− 0

Compare changes

Side-by-side
Inline

Show whitespace changes

Show one file at a time

Verified

b0184d80

production/web: Align archive instance with previous instance moma · b0184d80

Antoine R. Dumont authored 1 year ago

It's currently undersized to 5 workers (& 5 threads, the default values in the
template). In moma, it was configured with 32 workers with a timeout of 3600s. The
number of threads is kept to 5 and it is just explicited.

Refs. swh/infra/sysadm-environment#5110

swh/values/production/swh.yaml

+ 10

− 6

@@ -1151,8 +1151,17 @@ web:
  deployments:
    # Main instance to serve archive.softwareheritage.org
    archive:
      replicas: 4
      gunicorn:
        workers: 4
        threads: 2
        timeout: 3600
      # autoScaling:
      #   minReplicaCount: 2
      #   maxReplicaCount: 4
      #   cpuPercentageUsage: 100
      requestedCpu: 350m
      requestedMemory: 3072Mi
      requestedMemory: 6144Mi
      refreshSavecodenowStatus:
        enabled: true
        concurrencyPolicy: Forbid
@@ -1164,11 +1173,6 @@ web:
        configurationRef: postgresqlSyncmailmapsConfiguration
        priorityClassName: frontend-rpc-workload
        cron: "15 * * * *"
      replicas: 2
      # autoScaling:
      #   minReplicaCount: 2
      #   maxReplicaCount: 4
      #   cpuPercentageUsage: 100
      hosts:
        - archive.softwareheritage.org
        - base.softwareheritage.org