[k8s staging] Some pods are regularly evicted or in error

kubectl --context archive-staging-rke2 get pods -o wide | grep -v -e Running -e Pending -e Completed                                                                   12:25:56
NAME                                          READY   STATUS                   RESTARTS      AGE     IP              NODE                                NOMINATED NODE   READINESS GATES
graphql-6dcb5c5b84-cclv5                      0/1     Evicted                  0             3m4s    <none>          rancher-node-staging-rke2-worker2   <none>           <none>
graphql-6dcb5c5b84-xjhb5                      0/1     Evicted                  0             5d1h    <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-all-585787d8cc-ldkg9                   0/1     ContainerStatusUnknown   0             10m     10.42.233.177   rancher-node-staging-rke2-worker2   <none>           <none>
lister-bower-654c4896d5-4xrr7                 0/1     Evicted                  0             46m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-bower-654c4896d5-wlwhr                 0/1     Evicted                  0             11m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-golang-5c546c757-qk69t                 0/1     Evicted                  0             24h     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-launchpad-548db5d5-d78bj               0/1     Error                    0             12m     10.42.18.240    rancher-node-staging-rke2-worker3   <none>           <none>
lister-launchpad-548db5d5-ndg5w               0/1     Evicted                  0             25m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-launchpad-548db5d5-nzjxh               0/1     Error                    0             11m     10.42.233.190   rancher-node-staging-rke2-worker2   <none>           <none>
lister-opam-579d4b4599-jhns8                  0/1     Evicted                  0             11m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-opam-579d4b4599-zqfr6                  0/1     Evicted                  0             44m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
lister-opam-697c57db55-fbwjt                  0/1     Evicted                  0             2d      <none>          rancher-node-staging-rke2-worker2   <none>           <none>
loader-addforgenow-78bf56fd45-f9tj6           0/1     Error                    0             18h     10.42.233.156   rancher-node-staging-rke2-worker2   <none>           <none>
loader-bzr-6b6679854-w9cqd                    0/1     Evicted                  0             10d     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
loader-debian-776b5f47ff-rbfgg                0/1     Evicted                  0             42m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
loader-pypi-66f95595cb-lvqr5                  0/1     Evicted                  0             58m     <none>          rancher-node-staging-rke2-worker2   <none>           <none>
loader-svn-64ff4d6d79-8dg77                   0/1     Evicted                  0             10d     <none>          rancher-node-staging-rke2-worker2   <none>           <none>

It seems related to disk pressure on the nodes:

40m         Warning   Evicted                      pod/loader-pypi-66f95595cb-lvqr5                                          The node was low on resource: ephemeral-storage. Container loaders was using 650560000, which exceeds its request of 0.

changed milestone to %Dynamic infrastructure [Roadmap - Tooling and infrastructure]

There are also these events logged in the node status:

│   Warning  EvictionThresholdMet     8m8s                 kubelet  Attempting to reclaim ephemeral-storage                                                                                                                                  │
│   Normal   NodeHasDiskPressure      7m58s                kubelet  Node rancher-node-staging-rke2-worker2 status is now: NodeHasDiskPressure                                                                                                │
│   Normal   NodeHasNoDiskPressure    3m3s (x3 over 8m9s)  kubelet  Node rancher-node-staging-rke2-worker2 status is now: NodeHasNoDiskPressure                                                                                              │
│   Warning  FreeDiskSpaceFailed      28s                  kubelet  failed to garbage collect required amount of images. Wanted to free 13666077900 bytes, but freed 3289480395 bytes                                                        │
│   Warning  ImageGCFailed            28s                  kubelet  failed to garbage collect required amount of images. Wanted to free 13666077900 bytes, but freed 3289480395 bytes

assigned to @vsellier

added state:wip label

With the reservation of ~17Go of swap on the zfs dataset, it seems the nodes don't have enough space to load all the images needed by the pods:

The nodes are evicted from the cluster and recovering one by one explaining the disk space flip-flop.

Pompidou still have a lot of space on the scratch disk (300Go consumed on 750Go available)

I will increase the allocated space of the nodes on the scratch disks from 50Go to 100Go to see how the behavior will change.

Disk extended:

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data  49.5G  17.4G  32.1G        -       50G    15%    35%  1.00x    ONLINE  -

# zpool set autoexpand=on data

# As it was not set before the disk resize
# need to force the extend
# zpool online -e data vdb

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data  99.5G  14.4G  85.1G        -         -     7%    14%  1.00x    ONLINE  -

mentioned in commit swh/infra/swh-sysadmin-provisioning@2d3965ab

removed state:wip label

The cluster looks stable so far, there is no more evicted pods.

The disk consumption looks stable:

Closing for now, we'll reopened if the image GC is still not working as expected

closed