Remove cgroup v1 support

[kannon92]

Enhancement Description

• One-line enhancement description (can be used as a release note): Formally move cgroup v1 into an unsupported state
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/5573-cgroup-v1-unsupported/README.md
• Discussion Link: Many places. No longer testing CgroupV1 with fedora-coreos images kubernetes#134142 (comment). Its also a future goal in the cgroup v1 maintenance mode KEP
• PRs by stage and milestone:
  • beta - v1.35
    • KEP (k/enhancements) update PR(s): [KEP-5573]: Remove cgroup v1 support #5574
    • Code (k/k) update PR(s): [KEP-5573] Set failCgroupV1 to true kubernetes#134298
    • Docs (k/website) update PR(s): [KEP-5573]: announce deprecation of cgroups v1 website#52736

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kannon92]

My proposal is to make the default for fail-cgroup-v1 to true.

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubelet/config/v1beta1/types.go#L917

[kannon92]

/sig node

cc @haircommander @SergeyKanzhelev @dchen1107

[haircommander]

I'm not positive we're ready for this. I think CRI-O may investigate dropping support for v1, but I don't know if kube should yet

[kannon92]

I'm not positive we're ready for this. I think CRI-O may investigate dropping support for v1, but I don't know if kube should yet

I'm not sure we can really even support it from CRI-O side on latest OS.

Systemd has announced the removal of cgroup v1 code in v258. Fedora 43 will intake this so I think essentially we are going to have to drop testing of it.

    * Support for cgroup v1 ('legacy' and 'hybrid' hierarchies) has been
      removed. cgroup v2 ('unified' hierarchy) will always be mounted
      during system bootup and systemd-nspawn container initialization.

https://github.com/systemd/systemd/releases/tag/v258

Unless we pin systemd to older versions so we can test.

[kannon92]

We already removed testing on CRI-O it so to me that means its essentially unsupported.

[tzneal]

Some more data from the ecosystem, the AWS EKS AL2 AMI was the only one that still supported cgroup v1. AWS stopped shipping EKS AL2 AMIs after EKS 1.32, and that AMI will stop being shipped for all AWS supported Kubernetes versions in November.

The other two supported AWS EKS AMIs, AL2023 and Bottlerocket both use cgroup v2.

[haircommander]

I'm not sure we can really even support it from CRI-O side on latest OS.

CRI-O runs on other OS's, and still runs on it until we drop the code for it. But we can retain kube support longer than CRI-O, and the CRI-O piece is orthoganal to this KEP

[kannon92]

CRI-O runs on other OS's, and still runs on it until we drop the code for it. But we can retain kube support longer than CRI-O, and the CRI-O piece is orthoganal to this KEP

Yea I guess the main issue is who maintains that support. The PR is proposing making kubelet not start on a cgroup v1 node by default. It seems that the only way to support this is to pin OS to versions where systemd didn't drop support.

I worry that we have another 3-6 months of possible support on latest OSes (Fedora 43 will drop systemd support and that is coming soon). So if we don't announce anything we may just have to silently remove test lanes.

So basically our options:

• not do anything and let our CI blow up with failures once the OSes drop support
  • Remove testing lanes that start failing because we can't actually fix which linux kernel community members have removed.
• Maintain a OS with a pinned version of systemd (systemd 256) that we know will still work with cgroup v1 for downstream vendors who still offer cgroup v1 support

[SergeyKanzhelev]

I am all for removing the cgroupv1 support completely. Since we didn't announce deprecation officially: https://kubernetes.io/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/ we should do it ASAP.

I think the plan of:

• Disable cgroupv1 by default in 1.35
• Declare deprecation of cgroupv1 support officially
• Wait for two releases and remove all cgroupv1 related code

It is almost a year of keeping the cgroupv1 around. There still will be some OSes to test on during this time.

[SergeyKanzhelev]

Perhaps this can be tracked as a KEP to remove cgroupv1? alpha in 1.35 will be to mark deprecated and stable will be support completely removed. It may be more natural way to do it and we will have a KEP around for comments and attribution of work. @kannon92 are you ok with this?

[hakman]

We just removed support for cgroupv1 in kOps some weeks ago, for the same reason.
@neolit123 @carlory do you know what the cgroupv1 support status is in kubeadm?