Skip to content

Commit 3de6a5f

Browse files
alex-hunt-materializeclaudedef-
authored
Bump k8s-openapi feature to Kubernetes 1.34 (MaterializeInc#38059)
Bump k8s-openapi feature to Kubernetes 1.34. Bump kind to v0.32.0 and kubectl to v1.34.10 in the CI builder image and dev tooling. Bump all kind cluster configs to `kindest/node:v1.34.8` nodes (the pre-built 1.34 image for kind v0.32.0). Fix a race in our minio setup, which shows up more consistently in Kubernetes 1.34, where the PVC could end up on a different node and minio pods would never become ready. [DB-143](https://linear.app/materializeinc/issue/DB-143) Kubernetes 1.34 is the oldest currently supported version. Newer Kubernetes versions may not support legacy APIs over time, so better to stay current. This will require a similar bump in the cloud repo when bumping the submodule. ### Motivation Keep up to date, on supported platforms. ### Verification Nightly pipelines that exercise the bumped kind/kubectl/node images and should be run on this PR: - `cloudtest`, `cloudtest-slow`, `cloudtest-upgrade`: run kind via the rebuilt ci-builder image. - `orchestratord-*`: run in the `test/orchestratord` kind cluster, now on 1.34.8 nodes. - `k8s-node-recovery-*`: use `misc/kind/cluster-node-recovery-test.yaml`, now on 1.34.8 nodes. These are the steps the minio fix targets. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Dennis Felsing <dennis@felsing.org>
1 parent fd1dd6e commit 3de6a5f

11 files changed

Lines changed: 78 additions & 64 deletions

File tree

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,7 @@ jemalloc_pprof = "0.8.2"
399399
jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
400400
junit-report = "0.8.3"
401401
k8s-controller = "0.12.0"
402-
k8s-openapi = { version = "0.27.0", features = ["schemars", "v1_32"] }
402+
k8s-openapi = { version = "0.27.0", features = ["schemars", "v1_34"] }
403403
kube = { version = "3.1.0", default-features = false, features = ["client", "derive", "openssl-tls", "runtime", "ws"] }
404404
launchdarkly-server-sdk = { version = "3.1.1", default-features = false, features = ["hyper-rustls-native-roots", "crypto-aws-lc-rs"] }
405405
launchdarkly-sdk-transport = "0.1.4"

‎ci/builder/Dockerfile‎

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -325,16 +325,15 @@ RUN if [ $ARCH_GCC = x86_64 ]; then \
325325

326326
# Install KinD, kubectl, helm & helm-docs
327327

328-
# TODO(def-) Upgrading kind/kubectl seems to cause Cloudtest failures
329-
RUN curl -fsSL https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-$ARCH_GO > /usr/local/bin/kind \
328+
RUN curl -fsSL https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-$ARCH_GO > /usr/local/bin/kind \
330329
&& chmod +x /usr/local/bin/kind \
331-
&& if [ $ARCH_GO = amd64 ]; then echo 'c72eda46430f065fb45c5f70e7c957cc9209402ef309294821978677c8fb3284 /usr/local/bin/kind' | sha256sum --check; fi \
332-
&& if [ $ARCH_GO = arm64 ]; then echo '03d45095dbd9cc1689f179a3e5e5da24b77c2d1b257d7645abf1b4174bebcf2a /usr/local/bin/kind' | sha256sum --check; fi
330+
&& if [ $ARCH_GO = amd64 ]; then echo '50030de23cf40a18505f20426f6a8506bedf13c6e509244bd1fa9463721b0f54 /usr/local/bin/kind' | sha256sum --check; fi \
331+
&& if [ $ARCH_GO = arm64 ]; then echo 'b92cd615e97585de8ddade28ed5cd7feb4248d717c233eea5b03c37298900f5d /usr/local/bin/kind' | sha256sum --check; fi
333332

334-
RUN curl -fsSL https://dl.k8s.io/release/v1.24.3/bin/linux/$ARCH_GO/kubectl > /usr/local/bin/kubectl \
333+
RUN curl -fsSL https://dl.k8s.io/release/v1.34.10/bin/linux/$ARCH_GO/kubectl > /usr/local/bin/kubectl \
335334
&& chmod +x /usr/local/bin/kubectl \
336-
&& if [ $ARCH_GO = amd64 ]; then echo '8a45348bdaf81d46caf1706c8bf95b3f431150554f47d444ffde89e8cdd712c1 /usr/local/bin/kubectl' | sha256sum --check; fi \
337-
&& if [ $ARCH_GO = arm64 ]; then echo 'bdad4d3063ddb7bfa5ecf17fb8b029d5d81d7d4ea1650e4369aafa13ed97149a /usr/local/bin/kubectl' | sha256sum --check; fi
335+
&& if [ $ARCH_GO = amd64 ]; then echo '95bd70842bd11a524d24acd5b68726899e3488e153e45e2b4ae846545beda050 /usr/local/bin/kubectl' | sha256sum --check; fi \
336+
&& if [ $ARCH_GO = arm64 ]; then echo '52d3aeefea32fdfa3671ccd636be5da463ddfd0a2fc09d7bcbaedcff4c76cad5 /usr/local/bin/kubectl' | sha256sum --check; fi
338337

339338
RUN curl -fsSL https://get.helm.sh/helm-v4.0.0-linux-$ARCH_GO.tar.gz > helm.tar.gz \
340339
&& if [ $ARCH_GO = amd64 ]; then echo 'c77e9e7c1cc96e066bd240d190d1beed9a6b08060b2043ef0862c4f865eca08f helm.tar.gz' | sha256sum --check; fi \
@@ -517,16 +516,15 @@ RUN ln -s ../lib/node_modules/corepack/dist/corepack.js /usr/local/bin/corepack
517516
&& ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/local/bin/npx \
518517
&& corepack enable
519518

520-
# TODO(def-) Upgrading kind/kubectl seems to cause Cloudtest failures
521-
RUN curl -fsSL https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-$ARCH_GO > /usr/local/bin/kind \
519+
RUN curl -fsSL https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-$ARCH_GO > /usr/local/bin/kind \
522520
&& chmod +x /usr/local/bin/kind \
523-
&& if [ $ARCH_GO = amd64 ]; then echo 'c72eda46430f065fb45c5f70e7c957cc9209402ef309294821978677c8fb3284 /usr/local/bin/kind' | sha256sum --check; fi \
524-
&& if [ $ARCH_GO = arm64 ]; then echo '03d45095dbd9cc1689f179a3e5e5da24b77c2d1b257d7645abf1b4174bebcf2a /usr/local/bin/kind' | sha256sum --check; fi
521+
&& if [ $ARCH_GO = amd64 ]; then echo '50030de23cf40a18505f20426f6a8506bedf13c6e509244bd1fa9463721b0f54 /usr/local/bin/kind' | sha256sum --check; fi \
522+
&& if [ $ARCH_GO = arm64 ]; then echo 'b92cd615e97585de8ddade28ed5cd7feb4248d717c233eea5b03c37298900f5d /usr/local/bin/kind' | sha256sum --check; fi
525523

526-
RUN curl -fsSL https://dl.k8s.io/release/v1.24.3/bin/linux/$ARCH_GO/kubectl > /usr/local/bin/kubectl \
524+
RUN curl -fsSL https://dl.k8s.io/release/v1.34.10/bin/linux/$ARCH_GO/kubectl > /usr/local/bin/kubectl \
527525
&& chmod +x /usr/local/bin/kubectl \
528-
&& if [ $ARCH_GO = amd64 ]; then echo '8a45348bdaf81d46caf1706c8bf95b3f431150554f47d444ffde89e8cdd712c1 /usr/local/bin/kubectl' | sha256sum --check; fi \
529-
&& if [ $ARCH_GO = arm64 ]; then echo 'bdad4d3063ddb7bfa5ecf17fb8b029d5d81d7d4ea1650e4369aafa13ed97149a /usr/local/bin/kubectl' | sha256sum --check; fi
526+
&& if [ $ARCH_GO = amd64 ]; then echo '95bd70842bd11a524d24acd5b68726899e3488e153e45e2b4ae846545beda050 /usr/local/bin/kubectl' | sha256sum --check; fi \
527+
&& if [ $ARCH_GO = arm64 ]; then echo '52d3aeefea32fdfa3671ccd636be5da463ddfd0a2fc09d7bcbaedcff4c76cad5 /usr/local/bin/kubectl' | sha256sum --check; fi
530528

531529
# Use Helm 3 (not 4) because the cloud repo's bin/kind-create uses relative
532530
# file paths for vendored charts, which Helm 4 rejects as invalid URLs.

‎console/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Install the [aws-cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-s
6767
Install some k8s utils:
6868

6969
```shell
70-
brew install kubectl@1.24 k9s kind
70+
brew install kubernetes-cli@1.34 k9s kind
7171
```
7272

7373
Run the commands below to configure your aws account and k8s contexts

‎doc/developer/cloudtest.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ official [`kubernetes`] Python library to control the Kubernetes cluster.
2929
On Linux, use:
3030
3131
```
32-
curl -fL https://dl.k8s.io/release/v1.34.5/bin/linux/amd64/kubectl > kubectl
32+
curl -fL https://dl.k8s.io/release/v1.34.10/bin/linux/amd64/kubectl > kubectl
3333
chmod +x kubectl
3434
sudo mv kubectl /usr/local/bin
3535
```
@@ -48,7 +48,7 @@ official [`kubernetes`] Python library to control the Kubernetes cluster.
4848
On Linux, use:
4949
5050
```
51-
curl -fL https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-amd64 > kind
51+
curl -fL https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64 > kind
5252
chmod +x kind
5353
sudo mv kind /usr/local/bin
5454
```

‎misc/kind/cluster-node-recovery-test.yaml‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ kubeadmConfigPatches:
2222
"service-node-port-range": "32000-32063"
2323
nodes:
2424
- role: control-plane
25-
image: kindest/node:v1.33.1
25+
image: kindest/node:v1.34.8
2626
extraPortMappings:
2727
- containerPort: 32000
2828
hostPort: 32000
@@ -154,60 +154,60 @@ nodes:
154154
hostPort: 32063
155155

156156
- role: worker
157-
image: kindest/node:v1.33.1
157+
image: kindest/node:v1.34.8
158158
labels:
159159
materialize.cloud/disk: true
160160
materialize.cloud/availability-zone: "1"
161161
topology.kubernetes.io/zone: "1"
162162
- role: worker
163-
image: kindest/node:v1.33.1
163+
image: kindest/node:v1.34.8
164164
labels:
165165
materialize.cloud/disk: true
166166
materialize.cloud/availability-zone: "1"
167167
topology.kubernetes.io/zone: "1"
168168
- role: worker
169-
image: kindest/node:v1.33.1
169+
image: kindest/node:v1.34.8
170170
labels:
171171
materialize.cloud/disk: true
172172
materialize.cloud/availability-zone: "2"
173173
topology.kubernetes.io/zone: "2"
174174
- role: worker
175-
image: kindest/node:v1.33.1
175+
image: kindest/node:v1.34.8
176176
labels:
177177
materialize.cloud/disk: true
178178
materialize.cloud/availability-zone: "2"
179179
topology.kubernetes.io/zone: "2"
180180
- role: worker
181-
image: kindest/node:v1.33.1
181+
image: kindest/node:v1.34.8
182182
labels:
183183
materialize.cloud/disk: true
184184
materialize.cloud/availability-zone: "3"
185185
topology.kubernetes.io/zone: "3"
186186
- role: worker
187-
image: kindest/node:v1.33.1
187+
image: kindest/node:v1.34.8
188188
labels:
189189
materialize.cloud/disk: true
190190
materialize.cloud/availability-zone: "3"
191191
topology.kubernetes.io/zone: "3"
192192

193193
# node for the `quickstart` cluster replica
194194
- role: worker
195-
image: kindest/node:v1.33.1
195+
image: kindest/node:v1.34.8
196196
labels:
197197
materialize.cloud/disk: true
198198
materialize.cloud/availability-zone: "quickstart"
199199
topology.kubernetes.io/zone: "quickstart"
200200

201201
# only envd (nodes will be tainted in the setup)
202202
- role: worker
203-
image: kindest/node:v1.33.1
203+
image: kindest/node:v1.34.8
204204
labels:
205205
materialize.cloud/disk: true
206206
environmentd: true
207207
materialize.cloud/availability-zone: "3"
208208
topology.kubernetes.io/zone: "3"
209209
- role: worker
210-
image: kindest/node:v1.33.1
210+
image: kindest/node:v1.34.8
211211
labels:
212212
materialize.cloud/disk: true
213213
environmentd: true
@@ -216,7 +216,7 @@ nodes:
216216

217217
# for supporting services
218218
- role: worker
219-
image: kindest/node:v1.33.1
219+
image: kindest/node:v1.34.8
220220
labels:
221221
supporting-services: true
222222
materialize.cloud/availability-zone: "3"

‎misc/kind/cluster.yaml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ kubeadmConfigPatches:
2222
"service-node-port-range": "32000-32063"
2323
nodes:
2424
- role: control-plane
25-
image: kindest/node:v1.31.6
25+
image: kindest/node:v1.34.8
2626
extraPortMappings:
2727
- containerPort: 32000
2828
hostPort: 32000
@@ -154,21 +154,21 @@ nodes:
154154
hostPort: 32063
155155

156156
- role: worker
157-
image: kindest/node:v1.31.6
157+
image: kindest/node:v1.34.8
158158
labels:
159159
materialize.cloud/disk: true
160160
materialize.cloud/swap: true
161161
materialize.cloud/availability-zone: "1"
162162
topology.kubernetes.io/zone: "1"
163163
- role: worker
164-
image: kindest/node:v1.31.6
164+
image: kindest/node:v1.34.8
165165
labels:
166166
materialize.cloud/disk: true
167167
materialize.cloud/swap: true
168168
materialize.cloud/availability-zone: "2"
169169
topology.kubernetes.io/zone: "2"
170170
- role: worker
171-
image: kindest/node:v1.31.6
171+
image: kindest/node:v1.34.8
172172
labels:
173173
materialize.cloud/disk: true
174174
materialize.cloud/swap: true
@@ -177,7 +177,7 @@ nodes:
177177

178178
# no-disk
179179
- role: worker
180-
image: kindest/node:v1.31.6
180+
image: kindest/node:v1.34.8
181181
labels:
182182
materialize.cloud/disk: false
183183
materialize.cloud/swap: false
@@ -186,14 +186,14 @@ nodes:
186186

187187
# others
188188
- role: worker
189-
image: kindest/node:v1.31.6
189+
image: kindest/node:v1.34.8
190190
labels:
191191
materialize.cloud/disk: true
192192
materialize.cloud/swap: true
193193
materialize.cloud/availability-zone: "3"
194194
topology.kubernetes.io/zone: "3"
195195
- role: worker
196-
image: kindest/node:v1.31.6
196+
image: kindest/node:v1.34.8
197197
labels:
198198
materialize.cloud/disk: true
199199
materialize.cloud/swap: true

‎misc/python/materialize/cloudtest/app/materialize_application.py‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -152,12 +152,20 @@ def wait_for_sql(self) -> None:
152152
wait(condition="condition=Ready", resource="pod/environmentd-0")
153153

154154
start = datetime.now()
155-
while datetime.now() - start < timedelta(seconds=300):
155+
while True:
156156
try:
157157
self.environmentd.sql("SELECT 1")
158158
break
159-
except InterfaceError as e:
160-
# Since we crash environmentd, we expect some errors that we swallow.
159+
except (InterfaceError, OSError) as e:
160+
if datetime.now() - start > timedelta(seconds=300):
161+
raise
162+
# Since we crash environmentd, we expect some errors that we
163+
# swallow. pg8000 wraps most connection failures in
164+
# InterfaceError, but raw socket errors from its SSL
165+
# negotiation (e.g. ConnectionResetError when the connection
166+
# is accepted by the port-forwarding proxy and then reset
167+
# because environmentd is not listening yet) leak through
168+
# unwrapped.
161169
LOGGER.info(f"SQL interface not ready, {e} while SELECT 1. Waiting...")
162170
time.sleep(2)
163171

‎misc/python/materialize/cloudtest/k8s/minio.py‎

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
# the Business Source License, use of this software will be governed
88
# by the Apache License, Version 2.0.
99

10+
import yaml
11+
1012
from materialize import MZ_ROOT
1113
from materialize.cloudtest import DEFAULT_K8S_NAMESPACE
1214
from materialize.cloudtest.k8s.api.k8s_resource import K8sResource
@@ -35,9 +37,8 @@ def create(self) -> None:
3537
"true",
3638
)
3739

38-
# the PVC will be created afterwards
3940
for yaml_file in [
40-
"minio-standalone-deployment",
41+
"minio-standalone-pvc",
4142
"minio-standalone-service",
4243
]:
4344
self.kubectl(
@@ -46,22 +47,20 @@ def create(self) -> None:
4647
str(MINIO_YAML_DIRECTORY / f"{yaml_file}.yaml"),
4748
)
4849

49-
if self.apply_node_selectors:
50-
self.kubectl(
51-
"patch",
52-
"deployment",
53-
"minio-deployment",
54-
"--type",
55-
"json",
56-
"-p",
57-
'[{"op": "add", "path": "/spec/template/spec/nodeSelector", "value": {"supporting-services": "true"} }]',
58-
)
59-
60-
# the PVC needs to be created after patching the deployment
50+
# NOTE: The deployment must carry its final nodeSelector before it is
51+
# created, so it is injected here rather than patched in afterwards.
52+
# The claim's storage class binds with WaitForFirstConsumer, so the
53+
# provisioner pins the volume to whichever node the scheduler picks for
54+
# the first pod that consumes the claim. A pod created without the
55+
# nodeSelector can drive that decision even if it is replaced moments
56+
# later, pinning the volume to a node the final pod may not run on. The
57+
# pod then stays Pending forever, because nothing can satisfy both the
58+
# volume's node affinity and the pod's node selector.
6159
self.kubectl(
6260
"create",
6361
"-f",
64-
str(MINIO_YAML_DIRECTORY / "minio-standalone-pvc.yaml"),
62+
"-",
63+
input=self.deployment_manifest(),
6564
)
6665

6766
self.wait(
@@ -72,6 +71,17 @@ def create(self) -> None:
7271

7372
self.create_buckets(["persist", "copytos3", "copyfroms3"])
7473

74+
def deployment_manifest(self) -> str:
75+
with open(MINIO_YAML_DIRECTORY / "minio-standalone-deployment.yaml") as f:
76+
deployment = yaml.safe_load(f)
77+
78+
if self.apply_node_selectors:
79+
deployment["spec"]["template"]["spec"]["nodeSelector"] = {
80+
"supporting-services": "true"
81+
}
82+
83+
return yaml.dump(deployment)
84+
7585
def create_buckets(self, buckets: list[str]) -> None:
7686
cmds = [
7787
f"mc config host add myminio http://minio-service.{self.namespace()}:9000 minio minio123"

‎misc/scratch/provision.bash‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -79,12 +79,10 @@ rm -rf /tmp/awscli.zip /tmp/aws) &
7979
# uv
8080
(curl -fsSL https://astral.sh/uv/install.sh | sudo -u ubuntu sh >/dev/null 2>&1) &
8181

82-
# kubectl + kind + k9s. kind and kubectl mirror ci/builder/Dockerfile, which
83-
# deliberately holds them back: newer versions break Cloudtest (minio fails to
84-
# come up on the kind cluster).
85-
(curl -fsSL "https://dl.k8s.io/release/v1.24.3/bin/linux/$ARCH_GO/kubectl" -o /usr/local/bin/kubectl
82+
# kubectl + kind + k9s. kind and kubectl mirror ci/builder/Dockerfile.
83+
(curl -fsSL "https://dl.k8s.io/release/v1.34.10/bin/linux/$ARCH_GO/kubectl" -o /usr/local/bin/kubectl
8684
chmod +x /usr/local/bin/kubectl) &
87-
(curl -fsSL "https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-$ARCH_GO" -o /usr/local/bin/kind
85+
(curl -fsSL "https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-$ARCH_GO" -o /usr/local/bin/kind
8886
chmod +x /usr/local/bin/kind) &
8987
(curl -fsSL "https://github.com/derailed/k9s/releases/download/v0.50.18/k9s_Linux_$ARCH_GO.tar.gz" \
9088
| tar xzf - -C /usr/local/bin k9s

‎test/orchestratord/cluster.yaml.tmpl‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ kubeadmConfigPatches:
3030
"service-node-port-range": "32000-32063"
3131
nodes:
3232
- role: control-plane
33-
image: kindest/node:v1.32.5
33+
image: kindest/node:v1.34.8
3434
extraMounts:
3535
- containerPath: /var/lib/kubelet/config.json
3636
hostPath: "$DOCKER_CONFIG/config.json"
@@ -165,7 +165,7 @@ nodes:
165165
hostPort: 32063
166166

167167
- role: worker
168-
image: kindest/node:v1.32.5
168+
image: kindest/node:v1.34.8
169169
labels:
170170
materialize.cloud/swap: "true"
171171
materialize.cloud/availability-zone: "1"
@@ -175,7 +175,7 @@ nodes:
175175
- containerPath: /var/lib/kubelet/config.json
176176
hostPath: "$DOCKER_CONFIG/config.json"
177177
- role: worker
178-
image: kindest/node:v1.32.5
178+
image: kindest/node:v1.34.8
179179
labels:
180180
materialize.cloud/scratch-fs: "true"
181181
materialize.cloud/disk: "true"

0 commit comments

Comments
 (0)