Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ATMOSPHERE-263] feat: Support Kubernetes 1.29.x and 1.30.x #334

Merged
merged 11 commits into from
Aug 6, 2024
Merged

[ATMOSPHERE-263] feat: Support Kubernetes 1.29.x and 1.30.x #334

merged 11 commits into from
Aug 6, 2024

Conversation

0x00ace
Copy link
Contributor

@0x00ace 0x00ace commented Mar 19, 2024

Successfully booted Kubernetes 1.27.11 and upgraded to new Kubernetes versions 1.27.11 -> 1.28.7 -> 1.29.2 using commands:

 openstack coe cluster upgrade cluster_1 k8s-v1.28.7
 openstack coe cluster upgrade cluster_1 k8s-v1.29.2

Also, tested new images uploading to the custom container registry without any issues

mnaser
mnaser requested changes Mar 20, 2024
View reviewed changes
magnum_cluster_api/cmd/image_loader.py Outdated Show resolved Hide resolved
mnaser
mnaser requested changes Mar 21, 2024
View reviewed changes
Copy link
Member

@mnaser mnaser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, last around of changes:

  • We need to addthe 1.27.3 versions of images in image-loader

Once we have this, I will build and ship images for all these and switch the CI afterwards to include all the new images.

@0x00ace
Copy link
Contributor Author

0x00ace commented Mar 28, 2024

@mnaser I have all sets of the images added. Also, I have updated k8s versions.

@mnaser
Copy link
Member

mnaser commented Apr 10, 2024

@okozachenko1203 can you take into consideration your patch there?

@okozachenko1203 okozachenko1203 mentioned this pull request May 2, 2024
Closed
@KeplerSysAdmin
Copy link

Kepler Team here,
We would like to have this k8s version supported ASAP.
Thank you

@KeplerSysAdmin
Copy link

v1.27 end of life is 2024-06-28
When can we merge this PR?

@mnaser mnaser force-pushed the new_k8s_versions branch 4 times, most recently from 39feb2e to a45a0ee Compare June 7, 2024 17:46
@mnaser
Copy link
Member

mnaser commented Jun 7, 2024

recheck

@0x00ace
Copy link
Contributor Author

0x00ace commented Jun 12, 2024

@mnaser
I have successfully created a 1.28.x cluster and have upgraded it to 1.29.x
But to create a 1.29.x one needs to add a label keystone_auth_enabled=false

@okozachenko1203
Copy link
Member

recheck

@0x00ace
Copy link
Contributor Author

0x00ace commented Jun 18, 2024

For kubernetes >= 1.29.0 there is an issue with k8s-keystone-auth

Because of that issue, I need to pass a label keystone_auth_enabled=false

@0x00ace
Copy link
Contributor Author

0x00ace commented Jun 21, 2024

@mnaser
Should I also include Kubernetes 1.30.x support in this PR? The issue with k8s-keystone-auth I have mentioned above is also valid for 1.30.x. Everything else looks to be working.

@cgroschupp
Copy link

It might be a good idea to split this PR to get a newer k8s version than 1.27 as soon as possible, since the end of life is in 6 days. And then work on the keystone-auth problems in another PR and release it later.

@mnaser
Copy link
Member

mnaser commented Jun 25, 2024

I am struggling with finding a good way to manage the life cycle of these images other than "write a manual script" at this point. It seems like it's hard to find a way to get CI to react for changes, and the image builds are flakey, and the Sonobuoy tests are flakey too, and on top of it they eat a lot of resources...

@0x00ace
Copy link
Contributor Author

0x00ace commented Jun 28, 2024

@mnaser I will rework this PR to support only k8s 1.29 and 1.30
k8s 1.28 support in this PR

@mnaser
Copy link
Member

mnaser commented Jun 28, 2024

@0x00ace thanks, actually I figured out a nice way to build images that is far more reliable.

https://github.com/vexxhost/capo-image-elements

I am almost done with the CI bit and then we can ignore the image build here and ship the images from there to consume here.

@mnaser
Copy link
Member

mnaser commented Jun 28, 2024

Alright, 1.28 is in right now, I'm also trying to get image builds for EL9 working for 1.28, and then we can tackle this issue.

@0x00ace 0x00ace changed the title Support Kubernetes 1.28.x and 1.29.x Support Kubernetes 1.29.x and 1.30.x Jun 28, 2024
@0x00ace
Copy link
Contributor Author

0x00ace commented Jun 28, 2024

@mnaser
Updated PR for support k8s 1.29.x and 1.30.x
I'm using Rocky 9 with cilium and keystone_auth_enabled=false.
Cluster creation, resizing, and adding new nodegroups look to be working without issues.
Also, I have no issues with a cluster upgrade from 1.28 -> 1.29 -> 1.30.

@mnaser
Copy link
Member

mnaser commented Jun 28, 2024

vexxhost/capo-image-elements#4

This is what I'm struggling for EL9

mnaser and others added 2 commits August 2, 2024 13:59
@mnaser @okozachenko1203
Update versions to build cleanly
b8dab75
@okozachenko1203
use kustomize to enable keystone webhook after kubeadm init
7b87640 
it does not add the --authorization-* arguments until after kubeadm init run.
Once kubeadm init has finished, run kustomize to add the arguments to the kube-apiserver.yaml manifest.

ref: kubernetes/cloud-provider-openstack#2575
@okozachenko1203
Copy link
Member

/recheck

@okozachenko1203
append webhook authz mode only to avoid duplication with defaults
09d69dc 
api-server sets Node and RBAC as default authz modes in its command args.
And does not allow the mode specified more than once.
@okozachenko1203 okozachenko1203 changed the title Support Kubernetes 1.29.x and 1.30.x feat: Support Kubernetes 1.29.x and 1.30.x Aug 5, 2024
@okozachenko1203
Copy link
Member

image
1.29 and 1.30 passed for calico at least

@okozachenko1203
Copy link
Member

failed tests

1.29.6 cilium
[It] [sig-network] Services should serve endpoints on same port and different protocols [Conformance]

1.30 calico
[It] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should mutate configmap [Conformance]

1.30 cilium
[It] [sig-network] Services should serve endpoints on same port and different protocols [Conformance]

@okozachenko1203
Copy link
Member

okozachenko1203 commented Aug 6, 2024 •
edited
Loading

cilium/cilium#29913
cilium/cilium#9207

@okozachenko1203
Copy link
Member

image
all ubuntu tests passed

@okozachenko1203 okozachenko1203 requested review from mnaser and removed request for mnaser August 6, 2024 13:18
@KeplerSysAdmin
Copy link

@okozachenko1203
When is this version of magnum capi available?

@mnaser mnaser changed the title feat: Support Kubernetes 1.29.x and 1.30.x [ATMOSPHERE-263] feat: Support Kubernetes 1.29.x and 1.30.x Aug 6, 2024
@mnaser mnaser enabled auto-merge (squash) August 6, 2024 15:13
@mnaser mnaser disabled auto-merge August 6, 2024 15:13
@mnaser mnaser merged commit 761b592 into vexxhost:main Aug 6, 2024
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mnaser mnaser Awaiting requested review from mnaser

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@0x00ace @mnaser @KeplerSysAdmin @okozachenko1203 @cgroschupp