custom NodeConfig passed in via userData doesn't get propagated to the child launch templates that spin off of the parent

[jbilliau-rcd]

What happened:

Custom settings under kubelet in NodeConfig don't pass in or work. I'm passing in this as my user data into a custom launch template that I have my managed node groups configured to use:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--BOUNDARY
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: ${eks_cluster_name}
    apiServerEndpoint: ${api_server_endpoint}
    certificateAuthority: ${certificate_authority}
    cidr: 10.100.0.0/16
  kubelet:
    config:
      systemReserved:
        cpu: 150m

--BOUNDARY--

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

--==MYBOUNDARY==--

What you expected to happen:

Once I update my managed node groups to use that launch template, EKS then spins up a new, MNG-specific launch template based off that one.

However, as you can see, my custom config.systemReserved doesn't get passed in; instead EKS creates it's own config: block under kubelet: and just uses that. Thus, I have no way of setting custom settings for my systemReserved or kubeletReserved values, which I want to do to make our nodes a bit more resilient.

How to reproduce it (as minimally and precisely as possible):

Try passing in a custom block under kubelet....it doesn't carry through to the MNG-created child launch template even though it exists in the parent one created by you.

Environment:

• AWS Region: us-east-2
• Instance Type(s): EKS
• Cluster Kubernetes version: 1.30
• Node Kubernetes version: 1.30
• AMI Version: AL2023

[cartermckinnon]

Your MIME document doesn't look right to me. The NodeConfig is placed before the initial --==BOUNDARY==.

I assume you're using a launch template that doesn't include an AMI ID, so managed nodegroups is attempting to merge your user data with its own defaults. Something is probably going wrong in that process due to the syntax issues.

Managed nodegroups will add a MIME part that contains the bare-minimum NodeConfig (which includes your cluster details, you don't need to pass those yourself). Your launch template would only need a NodeConfig that looks like:

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      systemReserved:
        cpu: 150m

Surrounded by the proper MIME bits. If you want to open an AWS support case we can look into the specifics of your nodegroup 👍

[jbilliau-rcd]

ah ok....let me try tinkering with that then. So it sounds like this information is no longer correct? https://aws.amazon.com/blogs/containers/amazon-eks-optimized-amazon-linux-2023-amis-now-available/

It flat out says you will now need to provide, sounds like that's no longer the case? AWS must've automated that part so the customer (me) no longer has to?

[cartermckinnon]

It flat out says you will now need to provide, sounds like that's no longer the case?

I'll try to get this blog post clarified -- this is referring to a managed nodegroup using a launch template that includes an AMI ID (we usually call this a "custom AMI nodegroup"). In that scenario, MNG does not attempt to merge anything into your user data, you're responsible for providing all the required bits. If your nodegroup doesn't use a launch template, or uses a launch template that doesn't contain an AMI ID, MNG will inject the cluster details into the user data for you.

[jbilliau-rcd]

ahhhhh ok....we use a custom launch template (due to the need for user data, we install some custom agents), but we do NOT specify an AMI ID, so sounds like it will do all of the NodeConfig for us. Nice!

Also, I am now passing this in as my user data:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

--==MYBOUNDARY==
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      systemReserved:
        cpu: 150m

--==MYBOUNDARY==--

I just checked the managed node group launch template, and it looks like my change made it's way in there!

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    apiServerEndpoint: https://FC0503F4189116EA7776A51E97E18EEA.gr7.us-east-2.eks.amazonaws.com
    certificateAuthority: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJR1NIb1d3Vkw4K2d3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRFd01ESXlNRFV3TVRoYUZ3MHpOREE1TXpBeU1EVTFNVGhhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURna3FqRUI2Nmo3VHFWOGJtbnJKcjJQaCtnSVBjVm9ZZlJqeEVkTGJqbVdEcnQzTzQxRk4waWlCblkKTENGZVdCc0wvemVOTUxxRTM5ZGhQOGRhb2xJci84NmdBMkx5a2lTcXFUVUZZaHZpN1lOcVFnKytPZ0FxMC92SAoreEZJY3ZiWEg1bHBRQURJNHJmNkI3Y1psdTcyQUhCTWRzTWwremVFbTdsQ3NaakN1cW00NFQ1ZnAybUR0L1RQCmtEQ0FLVktkMUN5TmUzMUVPekpDdWhEbEM2a2ovRWRHK1h6M3BDQW9ieGR6WndNOGFXVmp3TDBNWmtqNFBrMUkKOWRRZlpyTUJjRUs4WWsydENINWRjY3o5NC9CVEoxR3FnM05LNFlvM29QQXZCN3c4Y0hCeGVRSG9kSDJpNzJUawozbmk2RjFyMk1PQ2tJNjJzRjdLcXpIRFVSL1hMQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRalZWV0RNZDRpVk5MOEtsaldMV2FkMU9ub0dUQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQXlIMmprUWZtRQozZ1dJV1g5Q2tHUHY0UnBocy83YzBucWl1bUFZY1pab3RXY2pQakpjSDUzWjRIb3VvVFNXbDJkcXZVM1hrTi9qCk1WeDNWWHZ1UjJIUkczdVUrc2t4YmZBclp6NUxnc3JPK0FUN2hOby83S3YzQlBKWFprbkpiU0ZjVWdrWWp2aE0KaUhVQk9EbUptN1cvUWxGL0J0YU5rOVZRVE5NRkFyNHh3Y0luUElSbDR5b0RBdGl2MFZqVE14K0MxeE5FREJraAp5Z1g0bHFDSVQ5bFNRK0Ztc3YrcTJBKysvM1FWUTM4V0Nlb3YzYk0rVjZMRU03dm1ubndiR2dENlAvSnBtQTBWCnIrNmh0am9vZ2lTK3VoNldrb2tyR1Z6RURhaDVNUHdZbEtpeHhHK1RDQXRzWWxORE5wMzZmUXZJUDVOTG44cVQKRVhLWXY5cXBIdnJrCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    cidr: 10.100.0.0/16
    name: tester-dev-418023852230-us-east-2
  kubelet:
    config:
      maxPods: 58
      clusterDNS:
      - 10.100.0.10
    flags:
    - "--node-labels=eks.amazonaws.com/nodegroup-image=ami-0747f67a5ac4d02ca,eks.amazonaws.com/sourceLaunchTemplateVersion=13,eks.amazonaws.com/capacityType=SPOT,eks.amazonaws.com/nodegroup=spot-tester2024100221022486470000000e,eks.amazonaws.com/sourceLaunchTemplateId=lt-0b87db9ee899cedaf"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash

sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      kubeReserved:
        cpu: 150m

--//--

Is it supposed to be a separate section at the bottom and not inline merged in? Or does EKS do that all in the background when it initializes a node?

[jbilliau-rcd]

looks like it didn't take effect :(

[jbilliau-rcd]

Hmmm....found these Karpenter docs, which we aren't using but it goes over AL2023 config. So my setup and behavior matches all of this.

https://karpenter.sh/docs/concepts/nodeclasses/#al2023-3

So not understanding why it's not applying....I'll keep digging.

[jbilliau-rcd]

@cartermckinnon so if the child MNG launch template has the correct user data (see above, matches what it should look like if you check the karpenter docs I linked), it is possible this merge is broke? My changes are not making their way into the kubelet config...I even tried other settings, like maxPods: 42 and that didn't work either. So I've gotten farther and corrected my original issue, but it still doesn't seem like it's working. Any other ideas?

[cartermckinnon]

Is it supposed to be a separate section at the bottom and not inline merged in?

Good question! The NodeConfig-s are merged by nodeadm at runtime on your node.

I think what's probably happening here is nodeadm is clobbering your kubeReserved setting specifically. Do you have the same issue if you set a field that EKS does not (e.g. systemReserved)?

amazon-eks-ami/nodeadm/internal/kubelet/config.go

Lines 333 to 340 in 9c6b810

	if cfg.Spec.Kubelet.Config != nil && len(cfg.Spec.Kubelet.Config) > 0 {
	mergedMap, err := util.Merge(kubeletConfig, cfg.Spec.Kubelet.Config, json.Marshal, json.Unmarshal)
	if err != nil {
	return err
	}
	if kubeletConfigBytes, err = json.MarshalIndent(mergedMap, "", strings.Repeat(" ", 4)); err != nil {
	return err
	}

[jbilliau-rcd]

nope, I tried setting systemReserved. Resulting MNG launch template:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    apiServerEndpoint: https://XXXXXXXXXX.gr7.us-east-2.eks.amazonaws.com
    certificateAuthority: XXXX
    cidr: 10.100.0.0/16
    name: XXXXXXX
  kubelet:
    config:
      maxPods: 58
      clusterDNS:
      - 10.100.0.10
    flags:
    - "--node-labels=eks.amazonaws.com/nodegroup-image=ami-0747f67a5ac4d02ca,eks.amazonaws.com/sourceLaunchTemplateVersion=17,eks.amazonaws.com/capacityType=SPOT,eks.amazonaws.com/nodegroup=spot-tester2024100221022486470000000e,eks.amazonaws.com/sourceLaunchTemplateId=lt-0b87db9ee899cedaf"

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  kubelet:
    config:
      systemReserved:
        cpu: 150m

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash

aws s3 cp s3://rc-tooling-prod/Qualys_RPM_Agent.rpm /tmp/
sudo rpm -ivh /tmp/Qualys_RPM_Agent.rpm
sudo /usr/local/qualys/cloud-agent/bin/qualys-cloud-agent.sh ActivationId=${qualys_activationid} CustomerId=${qualys_customerid} ServerUri=https://qagpublic.qg1.apps.qualys.com/CloudAgent/
${custom_nodegroup_userdata}

--//--

Check node's kubelet config file, nothing for systemReserved

I'm fairly certain this nodeConfig thing doesn't work....and yet, I find that hard to believe, as I can't be the first person to try this, lol....so not sure what to think.

[ndbaker1]

hey @jbilliau-rcd, since your node kubernetes version is 1.30, nodeadm is gonna take this alternative path to write the additional kubelet configuration to a drop-in file under /etc/kubernetes/kubelet/config.json.d/00-nodeadm.conf.

if you haven't already you probably want to validate the true config that kubelet's running with. There's a resource endpoint something like /configz for nodes IIRC

[cartermckinnon]

Thanks Nick! Completely slipped my mind that we used --config-dir on newer kubelets.

@jbilliau-rcd You can do something like this to check the effective config on a node:

kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/configz

[cartermckinnon]

@jbilliau-rcd let me know if you think there's still a problem, but I think we were just looking at the wrong config file. Should be working as expected after correcting the MIME syntax.

[jbilliau-rcd]

hey, yeah swamped past 2 days, I plan on revisiting this Monday and will post an update. Thanks!

[jbilliau-rcd]

You are correct, it is applying, both kubeReserved and systemReserved. Apologies, wasn't aware of that nodeadm conf file. Thanks @cartermckinnon and @ndbaker1 !