Remove predefined node taint after spegel pod finished initializing on the node

[shandr]

Describe the problem to be solved

On EKS, with Karpenter for nodes provisioning, in 99%, the workload pod starts before the spegel daemonset configures the node. This makes spegel useless in dynamic EKS + Karpenter environments. The suggested in the FAQ solution nidhogg is very unstable and not working as expected or not working at all. Additionally, with Karpenter, adding a taint that Karpenter is unaware of on the node can lead to a situation when Karpenter will create new nodes, because the pod doesn't have the required toleration.

Proposed solution to the problem

Use the "cilium" approach. With Karpenter (or some other methods) add startupTaints with some predefined name. When spegel initialized, it should remove the taint from the node.

[phillebaba]

I wish this was a solution provided in Kubernetes. There have been a couple of solutions proposed and none of them have been accepted. I wish that every project did not need to implement their own solution to fix this challenge.

I am for creating a solution that would work for both Karpenter users and Cluster Autoscaler users short term. Long term we should consider a KEP to solve this problem in Kubernetes.

[shandr]

With Cluster Autoscaler it's also possible to add some custom taint to Managed Nodegroups that spegel can remove after initialization. So it should work for both.

[Marvin0098]

@shandr We are seeing this exact same issue in our cluster's. Do you know any workaround besides using https://github.com/pelotech/nidhogg? (we really don't want to add yet a new tool to fix this issue). Doing it the way cilium does it would be perfectly fine for us.

Our current use case for spegel is that we have kubernetes jobs that run on a new node, to reduce the startup time we use spegel. But that does not work, because the pods are created before the spegel daemonset pod.