swh/staging: Deploy tmp emptyDir on afn loaders
Related to swh/infra/sysadm-environment#5464 (closed)
These modifications will deploy an emptyDir volume with medium
field set to Memory
on the deployment loader-add-forge-now
in staging environment.
The value limitedMemory
will be the limit of the pod's memory and the tmpfs volume space.
helm-diff
[swh] Comparing changes between branches production and staging_afn_emptydir (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra-next-version...
[swh] Generate config in staging_afn_emptydir branch for environment staging...
[swh] Generate config in staging_afn_emptydir branch for environment staging...
[swh] Generate config in staging_afn_emptydir branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace swh-cassandra-next-version...
[swh] Generate config in staging_afn_emptydir branch for environment production...
[swh] Generate config in staging_afn_emptydir branch for environment production...
[swh] Generate config in staging_afn_emptydir branch for environment production...
------------- diff for environment staging namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.yNa0WNdX/staging-swh.before, 141 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.yNa0WNdX/staging-swh.after, 141 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for environment staging namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.yNa0WNdX/staging-swh-cassandra.before, 459 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.yNa0WNdX/staging-swh-cassandra.after, 459 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned two differences
|___/
spec.template.spec.containers.loaders.resources (apps/v1/Deployment/swh-cassandra/loader-add-forge-now)
+ one map entry added:
limits:
memory: 2Gi
spec.template.spec.volumes.localstorage (apps/v1/Deployment/swh-cassandra/loader-add-forge-now)
- one map entry removed: + one map entry added:
ephemeral: emptyDir:
volumeClaimTemplate: medium: Memory
metadata: sizeLimit: 2Gi
labels:
type: ephemeral-volume
spec:
resources:
requests:
storage: 100Gi
storageClassName: local-path
accessModes:
- ReadWriteOnce
------------- diff for environment staging namespace swh-cassandra-next-version -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.yNa0WNdX/staging-swh-cassandra-next-version.before, 370 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.yNa0WNdX/staging-swh-cassandra-next-version.after, 370 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for environment production namespace swh -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.yNa0WNdX/production-swh.before, 154 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.yNa0WNdX/production-swh.after, 154 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
------------- diff for environment production namespace swh-cassandra -------------
_ __ __
_| |_ _ / _|/ _| between /tmp/swh-chart.swh.yNa0WNdX/production-swh-cassandra.before, 481 documents
/ _' | | | | |_| |_ and /tmp/swh-chart.swh.yNa0WNdX/production-swh-cassandra.after, 481 documents
| (_| | |_| | _| _|
\__,_|\__, |_| |_| returned no differences
|___/
Filling a tmp disk
root@test-emptydir-76df847d79-hlvx6:~# df -hT /tmp
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 256M 7.0M 250M 3% /tmp
root@test-emptydir-76df847d79-hlvx6:~# wget -P /tmp https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.8.0-amd64-netinst.iso
--2024-11-13 11:27:56-- https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.8.0-amd64-netinst.iso
Resolving cdimage.debian.org (cdimage.debian.org)... 194.71.11.173, 194.71.11.163, 2001:6b0:19::163, ...
Connecting to cdimage.debian.org (cdimage.debian.org)|194.71.11.173|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://chuangtzu.ftp.acc.umu.se/debian-cd/current/amd64/iso-cd/debian-12.8.0-amd64-netinst.iso [following]
--2024-11-13 11:27:57-- https://chuangtzu.ftp.acc.umu.se/debian-cd/current/amd64/iso-cd/debian-12.8.0-amd64-netinst.iso
Resolving chuangtzu.ftp.acc.umu.se (chuangtzu.ftp.acc.umu.se)... 194.71.11.167, 2001:6b0:19::167
Connecting to chuangtzu.ftp.acc.umu.se (chuangtzu.ftp.acc.umu.se)|194.71.11.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 661651456 (631M) [application/x-iso9660-image]
Saving to: '/tmp/debian-12.8.0-amd64-netinst.iso'
debian-12.8.0-amd64-netinst.iso 39%[==========================================================> ] 249.07M 32.0MB/s in 7.9s
Cannot write to '/tmp/debian-12.8.0-amd64-netinst.iso' (No space left on device).
root@test-emptydir-76df847d79-hlvx6:~# df -hT /tmp
Filesystem Type Size Used Avail Use% Mounted on
tmpfs tmpfs 256M 256M 0 100% /tmp
Merge request reports
Activity
added activity::MRO label
assigned to @guillaume
149 149 mountPath: /etc/swh 150 150 - name: localstorage 151 151 mountPath: /tmp 152 {{ if $use_json_logger }} 152 {{- if $use_json_logger }} 153 153 - name: configuration-template 154 154 mountPath: /etc/swh/logging-configuration.yml 155 155 subPath: "logging-configuration.yml" 156 156 readOnly: true 157 {{ end }} 157 {{- end }} 158 158 volumes: 159 159 - name: localstorage 160 {{- if and $deployment_config.tmpEmptyDir $deployment_config.limitedMemory }} WDYT to open a dedicated variable on the loader configuration to avoid any hardocoded configuration?
Something like
or deployment_config.workingVolume $.Values.loader.workingVolumes $.Values.tmpEphemeralStorage.default
?(variable names at you convenience)
It allows the charts to remain as generic as possible and avoid any opinionated configuration
changed this line in version 2 of the diff
153 153 maxReplicaCount: 3 154 154 requestedMemory: 1Gi 155 155 requestedCpu: 200m 156 limitedMemory: 2Gi It should be at least configured to the max allowed packfile size which is 4GB (https://gitlab.softwareheritage.org/swh/devel/swh-loader-git/-/blob/master/swh/loader/git/loader.py#L200)
For AFN, given the fact it's often repository we've never visited and https://grafana.softwareheritage.org/goto/78ZOIEGNk?orgId=1
It's probably worth it to increase the emptydir size to 5Gi to have some margin and the global requested memory to 8 or 10Gi. At least at the beginning to know how it will behave and avoid any afn errors.
changed this line in version 2 of the diff
added 1 commit
- fa02d79f - swh/loaders: Add a tmp volume dedicated variable
added 1 commit
- ed56ed34 - swh/loaders: Add a tmp volume dedicated variable
added 1 commit
- a36e491b - swh/loaders: Add a tmp volume dedicated variable