Missing unpigz command on kube nodes
The new images on the next-version environment cannot be pulled due to this error:
Warning Failed 40m (x178 over 14h) kubelet Failed to pull image "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20250113.1": failed to pull and unpack image "container-registry.softwareheritage.org/swh/infra/swh-apps/scheduler:20250113.1": failed to extract layer sha256:217c390bacd6556f97aac51a1778c300970b70d03490279ddc55284b29f7daf4: failed to get stream processor for application/vnd.docker.image.rootfs.diff.tar.gzip: fork/exec /usr/bin/unpigz: no such file or directory: unknownThe problem is also present in production (manual test)
Warning Failed 4s kubelet Failed to pull image "alpine": failed to pull and unpack image "docker.io/library/alpine:latest": failed to extract layer sha256:a0904247e36a7726c03c71ee48f3e64462021c88dafeb13f37fdaf613b27f11c: failed to get stream processor for application/vnd.oci.image.layer.v1.tar+gzip: fork/exec /usr/bin/unpigz: no such file or directory: unknown │
│It smells like a regression after the debian upgrade
Activity
added priority:High label
Looks like https://github.com/containerd/containerd/issues/3824
I'm guessing pigz got removed after the upgrade (and after kubelet had started) for some reason?
-
That seems to be the case on rancher-node-staging-rke2-worker1.
From
/var/log/apt/history.log:Start-Date: 2025-01-09 06:20:50 Commandline: /usr/bin/unattended-upgrade Remove: pigz:amd64 (2.6-1) End-Date: 2025-01-09 06:20:51Seems that the upgrade process is missing an apt autoremove step to clean up unused dependencies (before the node is rebooted)
What would that have changed? Overall, with what you suggest, the only difference would be that pigz would be dropped by the autoremove step, so we'd have the same behavior, wouldn't we?
And I currently do indeed an autopurge step after the reboot of the node (to keep the previous kernel image around just in case).
-
As mentioned in the containerd issue, the existence of pigz is checked only when containerd starts up (seems a bit silly to me, but it is what it is...). If pigz gets autoremoved after containerd starts, then unpacking gzip layers starts failing because pigz isn't in PATH anymore.
Removing the package before the reboot would avoid this race condition (at the cost of failing to unpack layers before the reboot, but that's probably not an issue).
staging-worker1 got a bunch of packages autoremoved by the unattended-upgrade run of 2025-01-09 (the day after the upgrade) so I think the autoremove step hadn't been done?
-
As mentioned in the containerd issue, the existence of pigz is checked only when containerd starts up (seems a bit silly to me, but it is what it is...).
ok, I missed that. Thanks for clarifying.
staging-worker1 got a bunch of packages autoremoved by the unattended-upgrade run of 2025-01-09 (the day after the upgrade) so I think the autoremove step hadn't been done?
I should have done it since i have it in my check list (through autopurge, iirc, that does an autoremove plus drops dangling files if it can).
But like any manual step, i may have missed it in the end. The simplest solution would be that...
marked this issue as related to #5415 (closed)
mentioned in commit swh/infra/puppet/puppet-swh-site@f06648f3
-
adding pigz to the list of packages installed by puppet (it doesn't really hurt)
You did it in swh-site [1]
[1] swh/infra/puppet/puppet-swh-site@0b5a6a31
adding the missing autoremoval step to the upgrade process
It's been documented in the rancher node debian upgrade procedure [2]
[2] swh/devel/swh-docs@274a2b96
configuring unattended upgrades not to autoremove packages
You did it too [2]
[3] swh/infra/puppet/puppet-swh-site@f06648f3
So all boxes are checked, I think we can close this issue \o/