staging cluster is in rather high memory pressure

Stop worker nodes' work [1]
Bump memory on mgmt1 node (reboot)
Uncordon nodes [2]

[1]

$ for i in $(seq 1 4); do kubectl --context archive-staging-rke2 drain rancher-node-staging-rke2-worker$i --ignore-daemonsets --delete-local-data --force; done

[2]

for i in $(seq 1 4); do kubectl --context archive-staging-rke2 uncordon rancher-node-staging-rke2-worker$i; done

I've created the new node worker5 (registered in the inventory too). It's not registered in the cluster yet for some reason (i'll check a tad later why).

mentioned in commit swh-sysadmin-provisioning@8127c025

mentioned in commit swh-sysadmin-provisioning@7770441f

I've created the new node worker5 (registered in the inventory too). It's not registered in the cluster yet for some reason (i'll check a tad later why).

There are issues logged [1] in the rancher-system-agent.service.

May 19 11:17:42 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: W0519 11:17:42.856867   75612 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 5; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
May 19 11:17:58 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: W0519 11:17:58.235808   75612 reflector.go:443] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 9; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
May 19 11:18:46 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: W0519 11:18:46.057552   75612 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *v1.Secret: an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>") has prevented the request from succeeding (get secrets.meta.k8s.io)
May 19 11:18:46 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: I0519 11:18:46.057753   75612 trace.go:205] Trace[1616138287]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (19-May-2023 11:18:00.312) (total time: 45745ms):
May 19 11:18:46 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: Trace[1616138287]: ---"Objects listed" error:an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>") has prevented the request from succeeding (get secrets.meta.k8s.io) 45745ms (11:18:46.057)
May 19 11:18:46 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: Trace[1616138287]: [45.745525005s] [45.745525005s] END
May 19 11:18:46 rancher-node-staging-rke2-worker5 rancher-system-agent[75612]: E0519 11:18:46.057882   75612 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *v1.Secret: failed to list *v1.Secret: an error on the server ("<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>") has prevented the request from succeeding (get secrets.meta.k8s.io)

unstuck.

Added a specific configuration file [1]
Restarted the rke2-agent (it complained on overlayfs snapshotter problem). [2] [3]
Gave some extra labels to that new node [4] [1]

root@rancher-node-staging-rke2-worker5:~# cat /etc/rancher/rke2/config.yaml.d/50-snapshotter.yaml
{
  "snapshotter": "native"
}

[2]

root@rancher-node-staging-rke2-worker5:~# systemctl status rke2-agent.service

[3]

May 19 13:08:29 rancher-node-staging-rke2-worker5 rke2[76720]: time="2023-05-19T13:08:29Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/rk>

[4] added labels {replayer, loader-metadata, lister}

$ kubectl --context archive-staging-rke2 label --overwrite node rancher-node-staging-rke2-worker5 swh/replayer=true
node/rancher-node-staging-rke2-worker5 labeled
$ kubectl --context archive-staging-rke2 label --overwrite node rancher-node-staging-rke2-worker5 swh/loader-metadata=true
node/rancher-node-staging-rke2-worker5 labeled
$ kubectl --context archive-staging-rke2 label --overwrite node rancher-node-staging-rke2-worker5 swh/lister=true
node/rancher-node-staging-rke2-worker5 labeled

changed the description

mentioned in commit swh/infra/ci-cd/swh-charts@0874d1f5

mentioned in commit swh/infra/ci-cd/swh-charts@bb51ad12

mentioned in commit swh/infra/ci-cd/swh-charts@75904576

mentioned in commit swh/infra/ci-cd/swh-charts@d300d423

mentioned in commit swh/infra/ci-cd/swh-charts@db893282

mentioned in commit swh/infra/ci-cd/swh-charts@760a85eb

The staging cluster is no longer in warning mode regarding the memory usage.

[1] https://rancher.euwest.azure.internal.softwareheritage.org/k8s/clusters/c-m-9n5h9nrf/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/d/rancher-cluster-nodes-1/rancher-cluster-nodes?orgId=1&from=1684139940998&to=1684744740998

I've created a board out of the one in the rancher (parametric on environment and cluster name) [1].

[1] https://grafana.softwareheritage.org/goto/7MInZMQ4k?orgId=1

Another more detailled one [1]. The cluster seems in good shape enough. So I'll close this.

[1] https://grafana.softwareheritage.org/goto/MtWzaMQVz?orgId=1

closed

the swap volume of the node was not configured to use a "large" zfs swap dataset.

The /tmp is small and a couple of workers are failing with a no space left error:

root@rancher-node-staging-rke2-worker5:~# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           976M  976M  4.0K 100% /tmp

root@rancher-node-staging-rke2-worker5:~# zfs create -V 16G data/swap
root@rancher-node-staging-rke2-worker5:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
data          56.5G  39.9G       24K  /data
data/docker   37.5K  39.9G     37.5K  /var/lib/docker
data/kubelet  3.12M  39.9G     3.12M  /var/lib/kubelet
data/rancher  40.0G  39.9G     40.0G  /var/lib/rancher
data/swap     16.5G  56.4G       12K  -

root@rancher-node-staging-rke2-worker5:~# mkswap -L swap /dev/zvol/data/swap
Setting up swapspace version 1, size = 16 GiB (17179865088 bytes)
LABEL=swap, UUID=7d1d7f96-d8e5-469f-993a-5d0151a5df37

root@rancher-node-staging-rke2-worker5:~# echo "RESUME=none" > /etc/initramfs-tools/conf.d/resume

root@rancher-node-staging-rke2-worker5:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.10.0-23-amd64

root@rancher-node-staging-rke2-worker5:~# swapoff /dev/dm-1

root@rancher-node-staging-rke2-worker5:~# lvremove '/dev/base-template-vg/swap_1'
Do you really want to remove active logical volume base-template-vg/swap_1? [y/n]: y
  Logical volume "swap_1" successfully removed

kubectl --context archive-staging-rke2 drain rancher-node-staging-rke2-worker5 --ignore-daemonsets --delete-local-data --force

swap entry removed from /etc/fstab
reboot
uncordon the node

kubectl --context archive-staging-rke2 uncordon rancher-node-staging-rke2-worker5

more info on #4676 (closed)

staging cluster is in rather high memory pressure

Designs

Child items ...

Activity