Migrate the loaders from the tmpfs workaround to a real in memory emptydir

changed title from migrating the loader from tmpfs workaround to a real in memory emptydir to Migrate the loaders from the tmpfs workaround to a real in memory emptydir

assigned to @guillaume

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@cd468fe9

mentioned in commit swh/infra/ci-cd/swh-charts@efc4afb9

mentioned in commit swh/infra/ci-cd/swh-charts@1ebd7ca8

mentioned in merge request swh/infra/ci-cd/swh-charts!507 (merged)

mentioned in commit swh/infra/ci-cd/swh-charts@fa02d79f

mentioned in commit swh/infra/ci-cd/swh-charts@ed56ed34

mentioned in commit swh/infra/ci-cd/swh-charts@a36e491b

The new tmp volume are deployed on the add-forge-now loaders in staging environment.
Two afn requests have been processed successfully.
The /tmp partition is space limited:

ᐅ for afn in $(kbs get po -n swh-cassandra -l app=loader-add-forge-now | awk 'NR>1{print $1}')
do echo $afn ; kbs exec $afn -n swh-cassandra -c loaders -- df -hT /tmp;done
loader-add-forge-now-6696b45587-mlhj2
Filesystem     Type   Size  Used Avail Use% Mounted on
tmpfs          tmpfs  5.0G  4.0K  5.0G   1% /tmp
loader-add-forge-now-6696b45587-n8lvk
Filesystem     Type   Size  Used Avail Use% Mounted on
tmpfs          tmpfs  5.0G  4.0K  5.0G   1% /tmp
loader-add-forge-now-6696b45587-wlct4
Filesystem     Type   Size  Used Avail Use% Mounted on
tmpfs          tmpfs  5.0G  4.0K  5.0G   1% /tmp

mentioned in commit swh/infra/ci-cd/swh-charts@1cf54d2a

mentioned in commit swh/infra/ci-cd/swh-charts@50010c7c

mentioned in commit swh/infra/ci-cd/swh-charts@c0624418

mentioned in commit swh/infra/ci-cd/swh-charts@5d4087be

mentioned in commit swh/infra/ci-cd/swh-charts@a638ce27

mentioned in commit swh/infra/ci-cd/swh-charts@ba0dbf4e

mentioned in commit swh/infra/ci-cd/swh-charts@5af18ffc

mentioned in commit swh/infra/ci-cd/swh-charts@854e4a6a

Here is the loaders memory usage of the last 7 days in staging:

Here are the loaders configurations:

ᐅ awk 'NR==141||NR>140&&NR<489&&(/^\s{2,4}[a-z-]+:$/||
/limitedMemory|tmpEphemeralStorage|sizeLimit/||/^\s{6}enabled/)' \
swh/values/staging/swh-cassandra.yaml
loaders:
  deployments:
    add-forge-now:
      limitedMemory: 10Gi
      tmpEphemeralStorage:
          sizeLimit: 5Gi
    add-forge-now-slow:
      limitedMemory: 10Gi
      tmpEphemeralStorage:
          sizeLimit: 5Gi
    arch:
      enabled: false
[...]
    pubdev:
      enabled: false
    rpm:
      enabled: false
    save-bulk:
    svn:
      limitedMemory: 25Gi
      tmpEphemeralStorage:
          sizeLimit: 15Gi

I forgot to modify the save-bulk loader.

mentioned in commit swh/infra/ci-cd/swh-charts@7a285c19

Here are the loaders configurations:

ᐅ awk 'NR==141||NR>140&&NR<494&&(/^\s{2,4}[a-z-]+:$/||
/limitedMemory|tmpEphemeralStorage|sizeLimit/||/^\s{6}enabled/)' \
swh/values/staging/swh-cassandra.yaml
loaders:
  deployments:
    add-forge-now:
      limitedMemory: 10Gi
      tmpEphemeralStorage:
          sizeLimit: 5Gi
    add-forge-now-slow:
      limitedMemory: 10Gi
      tmpEphemeralStorage:
          sizeLimit: 5Gi
    arch:
      enabled: false
    aur:
      enabled: false
    archive:
      enabled: false
    bioconductor:
      enabled: false
    bzr:
      enabled: false
    cran:
      enabled: false
    cvs:
      limitedMemory: 8Gi
      tmpEphemeralStorage:
          sizeLimit: 5Gi
    content:
      enabled: false
    directory:
      enabled: false
    git:
      limitedMemory: 20Gi
      tmpEphemeralStorage:
          sizeLimit: 15Gi
    git-checkout:
      limitedMemory: 8Gi
      tmpEphemeralStorage:
          sizeLimit: 5Gi
    hg-checkout:
      enabled: false
    svn-export:
      enabled: false
      limitedMemory: 1Gi
    debian:
      enabled: false
    deposit:
      limitedMemory: 18Gi
      tmpEphemeralStorage:
          sizeLimit: 15Gi
    golang:
      enabled: false
    save-code-now:
      limitedMemory: 15Gi
      tmpEphemeralStorage:
          sizeLimit: 10Gi
    maven:
      enabled: false
    mercurial:
      enabled: false
    npm:
      enabled: false
    oneshot:
      limitedMemory: 25Gi
      tmpEphemeralStorage:
          sizeLimit: 15Gi
    opam:
      enabled: false
    pypi:
      enabled: false
    pubdev:
      enabled: false
    rpm:
      enabled: false
    save-bulk:
      limitedMemory: 15Gi
      tmpEphemeralStorage:
          sizeLimit: 10Gi
    svn:
      limitedMemory: 25Gi
      tmpEphemeralStorage:
          sizeLimit: 15Gi