Document breaking changes for serviceClusterIPRange and OpenStack with S3 state store

[ederst]

When upgrading from kOps 1.25/26 to 1.27/1.28 I experienced some breaking changes:

I relied on the calculation of the serviceClusterIPRange from nonMasqueradeCIDR which was removed in #15866. This resulted in an error with the certificate as the certificate for the API was still issued for the previous service IP. Setting the serviceClusterIPRange to the previously calculated one was doing the trick.

Also, when having clusters running on OpenStack with gossip and the state stored in S3, newly created nodes will not join the cluster, as the S3_* variables are not passed on to the nodes anymore, due to the change here. The attempt to fix this was closed. Could be that this affects other clouds as well, but I have not checked this.

So, if those changes are legit (and not bugs), then I'd say let's document them at least as breaking changes.

[k8s-ci-robot]

Hi @ederst. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zetaab]

/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zetaab

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [zetaab]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hakman]

/retest

[crypticophelia]

Having recently gone through the pain of upgrading to 1.28.1 caused by this issue, it would have been superb if perhaps this BREAKING CHANGE note had been cherry picked back into the release notes. It took us 3 broken clusters, and two rebuilds to finally figure out what shenanigans were occurring here, and then only by pure chance of this issue sifting to the top of the search results.

Never once did it occur to any of us to check the 1.29 alpha release notes for 1.28 breaking change information, and I cannot fathom a world where it would be normal to operate in that manner.

[ederst]

@crypticophelia The documentation is only generated from the main branch, so at least the information is there (https://kops.sigs.k8s.io/releases/1.28-notes/). But yeah it can be misleading if information like this is not backported to the appropriate release branch and then only looking at the MD file in there.

@hakman @zetaab So I am not sure if this is an oversight or just a "flaw" in how documentation is treated (no cherry picks necessary due to gen from main branch).