📖 ✨ 🧑‍🤝‍🧑 add proposal for Node Bootstrapping working group

[t-lo]

What this PR does / why we need it:

Propose a working group for node bootstrapping and cluster provisioning.
The need for this working group originated from an ongoing discussion around separating cluster provisioning and node bootstrapping, as stated in the WG's User Story.

Which issue(s) this PR fixes

• Cleaner separation of kubeadm and machine bootstrapping #5294
• Ignition v3.x support #9157

CC

• "@elmiko"
• "@eljohnson92"
• "@johananl"
• "@tormath1"

Tags

/area provider/bootstrap-kubeadm
/area bootstrap

/kind documentation
/kind proposal

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sbueringer for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @t-lo. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[t-lo]

"@elmiko" "@eljohnson92" I took the liberty to add you as co-stakeholders to the WG proposal - IIRC you expressed interest in participating. I hope that's OK?

[t-lo]

Thank you Johanan, Fabrizio, and Stefan for tuning in! This is immensely helpful.
I also love the fact that we're already iterating over design thoughts; this is exactly the momentum we were hoping for with the working group.

Made a few changes to the proposal; reworked the whole user story part to focus on goals instead of implementations, and rephrased the "problem statement" section a bit to not hint at a solution when describing the issue.

Added a new section on stability and compatibility - this really was the proverbial elephant in the room for me since in Flatcar, we put a massive (occasionally painful) focus on never breaking user workloads ever - kudos to Stefan for calling this out. I'll make sure to hold the working group proposals to an equally high standard.

I don't think we're quite there yet but we're definitely making progress. Ready for another round of feedback!

[t-lo]

@sbueringer , @fabriziopandini what do you think? Could you give it another pass?

[johananl]

Thanks! Added a few comments.

[fabriziopandini]

I will try to come back to this after code freeze next week, I need to focus on stuff to get merged + CI signal and bandwidth is limited 😢

[chrischdi]

Also showing up, I will need some more time to read myself into all of this but my focus now is first the upcoming CAPI release!

[t-lo]

Rebased to latest main, squashed to a single commit, and added the meeting cadence and link to the meeting calendar.

@schrej could you please have another look?

[schrej]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 74de9f021d75cde603c0e2f0b71ea274de02f5f5

[johananl]

/ok-to-test

[johananl]

@t-lo the CI doesn't like the link to the calendar file. I wonder if it's OK to ignore this test.

[rothgar]

I'm a bit confused by the problem statement in the document.

1. Is this intended to only apply to configuration after the OS has booted and been installed? Cloud-init and ignition make a lot of assumptions about what's on the node and cannot be abstracted.
2. Is this only intended to be used for kubeadm based provisioning? I know it mentions other options but the focus and ultimate implementation is based on kubeadm with others being optional
3. Is this meant to re-configure nodes after they've been provisioned? From my understanding this is mostly designed to configure a node from a base, unconfigured state. Ignition and cloud-init don't re-configure or modify nodes and the only option is to destroy/replace

I'm trying to figure out if I should be involved at all (right now it seems like the answer is no) because Talos Linux doesn't use ignition, cloud-init, or kubeadm to configure (and re-configure) a node for kubernetes.

[johananl]

Hey @rothgar.

This working group is about improving the separation between bootstrap and provisioning. "Bootstrap" means "turning a server into a k8s node". "Provisioning" means "running cloud-init/Ignition to customize a server". "Server" in this context means a physical bare metal server or a VM.

We don't intend to alter the functionality of bootstrap or provisioning in this working group, we're concerned mainly with how we can maintain and evolve multiple provisioner implementations (e.g. cloud-init, Ignition) over time, which hopefully requires only under-the-hood changes with no user impact. This does entail touching CABPK (the kubeadm bootstrap provider) because right now bootstrap and provisioning are tightly coupled, which means we can't touch e.g. cloud-init code without touching kubeadm-related code.

We hope to get to an acceptable state in CABPK which could serve as a "template" for other bootstrap providers, however there's still a way to go before we get there. We want to avoid making API changes to the extent that's possible, but of course if we do end up proposing API changes then this would affect all relevant providers, including Talos.

I suggest you keep an eye on the products of this working group to determine if/when you need to be involved (we will provide updates in the CAPI office hours so you can watch the notes/recordings), however I think that at the moment this work is irrelevant for anything which doesn't use kubeadm, cloud-init or Ignition. We'll be sure to broadcast any significant changes loudly to the community in case we realize they're inevitable.

[t-lo]

I'd like to keep the iCal link - it's very convenient to be able to import the meeting series. Here's the link, it works: https://calendar.google.com/calendar/ical/90d22cde4972f248d6516a96de05ef62553644fff261e2150f5f229546d59d41%40group.calendar.google.com/public/basic.ics Tested on a number of machines. If we could figure out what makes this test fail then that would be awesome...

@rothgar If you like to know more, feel free to join our WG office hours Thursday next week (Dec 19), at 5pm UTC - happy to chat and to give an intro to the whole thing.

[t-lo]

/ok-to-test

[johananl]

/retest

[t-lo]

/retest

[schrej]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 5aed488f3075793a52ddf47fa7eda64d36334ac0