Kubernetes Nodes: cgroup v1 Leaves in v1.38, Swap Triples Density, PVC Unused Goes Beta

Three Kubernetes posts from September and October touch the same layer, the node: the move off cgroup v1, node swap backed by local SSDs, and a new way to find idle PersistentVolumeClaims. Each has its own version timeline, so they are taken separately below.

cgroup v1: Kubelet Refuses It From v1.35, Removal Planned for v1.38 #

The cgroup v2 post by Paco Xu of DaoCloud sets out the timeline. cgroup v2 support has been stable since v1.25, and cgroup v1 entered maintenance mode in v1.31. Since v1.35, failCgroupV1 defaults to true, so the kubelet does not start on a cgroup v1 node unless an administrator sets failCgroupV1: false, and kubeadm's SystemVerification preflight check errors on cgroup v1 with kubelet v1.35 or later. The post states that the remaining cgroup v1 fallback is scheduled for removal in Kubernetes v1.38.

The cgroup documentation lists the requirements: kernel 5.8 or later, containerd 1.4 or CRI-O 1.20 or later, and the systemd cgroup driver for both the kubelet and the runtime. Software that reads the cgroup filesystem directly needs cgroup v2 support: cAdvisor 0.43.0 or later, OpenJDK 8u372, 11.0.16 or 15 and later, automaxprocs 1.5.1 or later, and Node.js 20.3.0 or later, since Node.js 18 may size its heap from the host's memory instead of the pod limit. On a node, stat -fc %T /sys/fs/cgroup/ prints cgroup2fs for cgroup v2 and tmpfs for v1.

The cgroup post also describes Memory QoS as it stood in v1.36, as an alpha feature with a default memoryThrottlingFactor of 0.9. That changed in v1.37: Memory QoS is beta and enabled by default, but the default throttling factor is now null, so no memory.high, memory.min or memory.low values are written unless configured. A kubelet configuration that relied on the old 0.9 default needs the value set explicitly to keep throttling.

Node Swap on Local SSD: Up to 3× Pod Density in Google's Benchmarks #

Node swap has been generally available since v1.34. The swap post by Ocean Xie and Yuan Wang reports benchmarks on GKE with swap on local SSD: a Linux kernel build ran with a 300 MB limit instead of 600 MB, finishing in 374 s against a 433 s baseline; headless Chrome went from 80 to 160 pods under gVisor and from 40 to 50 under Kata; plain runc containers went from 512 to 768 pods on a c4-standard-32 node; and Python sandboxes went from 80 to 240 sessions. The post reports density gains of up to 3×, "often with little or no latency cost".

The limits matter as much as the gains. Squeezing the kernel build to 200 MB pushed the active working set into swap and slowed it by more than 40%, and at peak density the latency increase came mainly from pods competing for CPU. Swap is configured in the kubelet:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
failSwapOn: false
memorySwap:
  swapBehavior: LimitedSwap

The post advises running workloads with Burstable QoS, with memory limits above requests, so the node can move idle memory to swap; separate swap accounting depends on cgroup v2.

PVC Unused Condition: Beta in v1.37 #

In v1.37 the PersistentVolumeClaimUnusedSinceTime feature gate is beta and enabled by default. The PVC protection controller adds an Unused condition to each PVC: True with reason NoPodsUsingPVC when no non-terminated pod references it, False with PodUsingPVC otherwise. Completed pods do not count, pending and even unschedulable pods do, and the condition's lastTransitionTime records when the PVC became idle, which makes queries such as "idle for more than 30 days" possible. The feature was alpha in v1.36.