feat(kubernetes): add v0.5.0 Helm chart delivery for CubeSandbox#779
feat(kubernetes): add v0.5.0 Helm chart delivery for CubeSandbox#779vimiix wants to merge 5 commits into
Conversation
Add a Helm chart under deploy/kubernetes/ so CubeSandbox v0.5.0 can be delivered on Kubernetes/TKE from an internal registry. deploy/kubernetes/chart/ - Helm chart pinned to appVersion 0.5.0. - Control-plane workloads: cube-master, cube-api, cubemastercli, cube-proxy-node, cube-webui, plus optional built-in mysql and redis StatefulSets. - Compute-side workloads: cube-node DaemonSet with cube-egress sidecar, cube-egress-net, cube-node-init, cube-pvm-host-bootstrap init containers, and node-local cube-dns. - Preflight validation (templates/validate.yaml) and Helm test hooks (templates/tests/) covering node-health and RBAC. - All Cube-owned images.*.repository default to ccr.ccs.tencentyun.com/cubesandbox-chart/*, tag v0.5.0. deploy/kubernetes/images/ - Role-specific Dockerfiles for cube-node, cube-node-init, cube-pvm-host-bootstrap, cube-webui, cube-egress-net, and cubemastercli. - build-cube-images.sh builds and optionally pushes all ten Cube images. cube-master, cube-api, cube-proxy-node, and cube-egress (plus its openresty tproxy base) are built from the in-tree CubeMaster/CubeAPI/CubeProxy/CubeEgress source. cube-node, cube-master, and cubemastercli additionally consume binaries extracted from the v0.5.0 one-click release bundle on SourceForge. - Downloads honor the v0.5.0 arch-suffixed layout via a new ONE_CLICK_ARCH knob (default amd64), which drives both the tarball URL and the extracted top-level directory name. - The curl invocation gates --retry-all-errors on runtime feature detection so the script also works on distributions shipping curl < 7.71. Default registry / tag: - REGISTRY=ccr.ccs.tencentyun.com/cubesandbox-chart - IMAGE_TAG=v0.5.0 Verified: - helm lint passes. - helm template with representative overrides renders every Cube-owned image as ccr.ccs.tencentyun.com/cubesandbox-chart/<name>:v0.5.0. - All ten images built and pushed successfully with the default REGISTRY and IMAGE_TAG.
![cubesandboxbot[bot]](https://avatars.githubusercontent.com/u/20101770?s=60&v=4){kind=small}
Comprehensive Review: PR #779 — CubeSandbox v0.5.0 Helm ChartThis is a well-structured, production-quality PR. The chart follows established Helm patterns, includes comprehensive preflight validation (~30 checks in validate.yaml), and the image build system has thoughtful features like source pinning via 🔴 Critical1. Egress MITM CA private key exposed to three components The egress root CA private key ( Fix: Mount the egress CA secret only to cube-egress. Use a subordinate CA or certificate-based scheme for cube-api and cube-master (they likely only need the public CA cert, not the private key). Shorten the CA validity to 365 days. 2. All resource requests/limits default to empty Every component's Fix: Set baseline requests/limits in values.yaml. e.g., cube-node: 🟠 High3. Default credentials should use generated passwords MySQL and Redis passwords are static defaults ( Fix: Use 4. No artifact integrity verification in build pipeline Three artifacts are downloaded via SourceForge HTTPS URLs with zero content verification — no SHA256 checksum, no GPG signature. A Fix: Download a 5. Node RBAC combined with SYS_PTRACE + hostPID enables node compromise The cube-node ClusterRole grants Fix: Restrict node RBAC to 6. CoreDNS Every DNS query on every compute node is logged. On nodes running hundreds of sandbox VMs, this generates gigabytes of log data per day. Fix: Replace 7. CoreDNS cache TTL (300s) mismatches A record TTL (60s) CoreDNS caches responses for 5 minutes even though the template A records advertise a 60-second TTL. When the sandbox gateway IP changes, clients get stale responses for up to 5 minutes. Fix: Remove the explicit cache argument to use per-record TTL: 8. Redis/MySQL PVCs may use wrong StorageClass The Fix: In values.yaml, add 🟡 Medium9. DNS config block duplicated across three templates The same 10-line Fix: Extract into a 10. ARCHITECTURE.md validation rules section is significantly incomplete Only 7 validation rules are listed, but Fix: Expand the rules section to cover all checks, or add a note that the complete set lives in 11. README.md missing The Fix: Add a note in the "Cube Node one-click parity" section describing the 12. pvm-host-bootstrap.sh sleeps 3600s after reboot request After requesting a host reboot, the init container sleeps for 1 hour before exiting. This keeps the DaemonSet pod stuck in Init state for an hour. Fix: Reduce to 120s grace period. If the host hasn't started rebooting by then, restart the pod. ✅ Positive highlights
|
Address inline review comments on the v0.5.0 Helm chart delivery. Chart: - templates/node-daemonset.yaml: fix CUBE_SANDBOX_DNS_SERVERS serialization. `join "," .` was joining every top-level context key (.Values, .Release, .Chart, .Files, ...) into the env var value, which produces garbage and can leak Values content. Use `.Values.cubeNode.dns.sandbox.nameservers` explicitly. - templates/tests/node-health.yaml: extend helm.sh/hook-delete-policy on all eight test Pods to `before-hook-creation,hook-succeeded` so successful test Pods are cleaned up automatically instead of lingering until the next `helm test` run. - templates/mysql.yaml: drop `--default-authentication-plugin=mysql_native_password`. The in-tree MySQL clients (go-sql-driver/mysql v1.9.3 for CubeMaster, sqlx 0.8 mysql for CubeAPI) both support MySQL 8's default caching_sha2_password plugin, so the SHA-1 override is no longer required. Image build script: - images/build-cube-images.sh: consolidate `build_image` and `build_component_image` into a single function with an optional dockerfile argument, dropping ~30 lines of near-duplicated code and switching all four callsites to the merged signature. - images/build-cube-images.sh: fail fast when the sandbox-package is missing a required cube-node component. Previously `copy_if_exists` silently skipped missing directories and a follow-up `mkdir -p` created empty ones, which the Dockerfile then COPYed into the image. Cubelet, network-agent, cube-shim, cube-kernel-scf, cube-image, cube-vs, cube-snapshot, and scripts/common are now required, and the same pattern is used for the cube-webui context. `copy_if_exists` is removed since it has no remaining callers. Verified: - helm lint deploy/kubernetes/chart passes. - helm template with cubeNode.dns.sandbox.nameservers ["8.8.8.8","1.1.1.1"] now renders `value: "8.8.8.8,1.1.1.1"` for CUBE_SANDBOX_DNS_SERVERS. - bash -n on build-cube-images.sh passes and no residual build_component_image / copy_if_exists references remain.
Add two operator-facing docs under deploy/kubernetes/chart/docs/ and link them from the chart README. - docs/QUICKSTART.md: end-to-end one-click install walkthrough covering cluster/node prerequisites, node label & taint setup, optional image build, minimal runtime-values.yaml sample, `helm upgrade --install` command, verification steps, single-node trial layout, compute-only mode with external control plane, and uninstall cleanup checklist. - docs/FAQ.md: common install and runtime issues grouped by subsystem (install/validate, node scheduling, control plane / database, cube-node & PVM kernel & sandbox runtime, CubeProxy / TLS / DNS, egress network, upgrade/rollback/uninstall, image build). Each entry documents the failure signal, root cause, and concrete kubectl / helm commands to unblock. - chart/README.md: list QUICKSTART.md and FAQ.md alongside ARCHITECTURE.md in the Directory section and a new Documentation section so first-time users can discover the guides directly from the chart README.
build-cube-images.sh compiled cube-api, cube-proxy-node, cube-egress,
and cube-egress-net directly from the current worktree's CubeMaster/,
CubeAPI/, CubeProxy/, and CubeEgress/ trees. When the worktree was
ahead of the release tag (e.g. building from the chart delivery
branch that carries additional CubeAPI features), the resulting
images drifted from the tag advertised in Chart.yaml / values.yaml.
In the v0.5.0 default build this manifested as cube-api requiring an
examples/ directory that only exists in commits merged after the
v0.5.0 tag, causing a startup FATAL when the same images were
deployed via this chart.
Introduce SOURCE_REF (default: ${VERSION}, i.e. v0.5.0) which
exports the four source trees at that git ref into
${BUILD_ROOT}/source-tree/ via `git archive` and points REPO_ROOT
there for the duration of the build. This keeps the delivered images
aligned with the release tag independent of the current chart branch.
- SOURCE_REF="" disables pinning and falls back to the current
worktree (useful for local development against unreleased changes).
- SOURCE_REF=<ref> targets any branch, tag, or commit SHA.
- The exported tree is cached; a stamp file avoids re-exporting when
the ref has not moved.
Also document the new knob and its intent in
deploy/kubernetes/images/README.md.
| {{ $placeholderCertKey | quote }}: {{ index $existing.data $placeholderCertKey | quote }} | ||
| {{ $placeholderKeyKey | quote }}: {{ index $existing.data $placeholderKeyKey | quote }} | ||
| {{- else }} | ||
| {{- $ca := genCA .Values.cubeEgress.ca.selfSigned.caCommonName (int .Values.cubeEgress.ca.selfSigned.caValidityDays) -}} |
There was a problem hiding this comment.
Security: Egress CA private key mounted to three components unnecessarily
genCA generates a root CA private key that enables TLS interception (MITM) for all outbound sandbox connections. This key is mounted to cube-egress, cube-api (api.yaml:l.86-89), and cube-master (master.yaml:l.75-79). Both cube-api and cube-master should only need the public CA certificate, not the signing key. A compromise of either component leaks the root signing key with 10-year validity, allowing decryption of all past/future sandbox egress TLS traffic.
Fix: Remove the egress CA volume mount from cube-api and cube-master, or mount only the public certificate. Consider shortening the CA validity period from 3650 days to 365.
There was a problem hiding this comment.
Acknowledged. Splitting the shared MITM CA into a subordinate CA per component (cube-egress signing subordinate, cube-api / cube-master trust-only public bundle) is an architectural change that touches CubeEgress key management, cube-egress-net iptables trust chain, and the CubeMaster/CubeAPI code paths that load the CA bundle today.
Leaving this thread open so the upstream discussion can drive the split; the chart will follow whichever key hierarchy the CubeSandbox core team standardizes on. Tracking-only for this PR.
| - apiGroups: ["coordination.k8s.io"] | ||
| resources: ["leases"] | ||
| verbs: ["get", "list", "create", "update", "patch"] | ||
| - apiGroups: [""] |
There was a problem hiding this comment.
Security: Node RBAC permissions are overly permissive
The cube-node ClusterRole grants patch/update on nodes (full node objects, not just /status), combined with SYS_PTRACE capability + hostPID: true in node-daemonset.yaml (l.28-29). A compromised cube-node container can ptrace-attach to kubelet on the host (stealing service account tokens) and then modify node taints/labels via the RBAC to schedule workloads on control-plane nodes.
Fix: Restrict to nodes/status with only get/patch verbs. Consider adding resourceNames: [] to limit which nodes can be modified. Evaluate whether SYS_PTRACE is actually required.
There was a problem hiding this comment.
Acknowledged. The cluster-scoped nodes/patch permission on the cube-node ServiceAccount pairs with hostPID + SYS_PTRACE on the runtime container, which together do enable the escalation path described.
Restricting the ClusterRole to nodes/status subresource and stripping SYS_PTRACE requires a coordinated change with Cubelet — Cubelet today writes NIC/label/taint annotations on the parent node object and uses ptrace-based sandbox introspection. That trade-off belongs in the CubeSandbox core project, not the chart.
Leaving this thread open so the upstream RBAC/capability tightening can drive the change; the chart will mirror whatever minimal permission set Cubelet needs after the split.
|
Test report：https://docs.qq.com/doc/DUWR1RnRteHp5ZVpX |
Address the remaining review comments on PR TencentCloud#779 (reliability, security, correctness, and code quality). Ranged from a one-line helper introduction to a full script extraction — all preserving default behavior when operators do not opt in to the new knobs. Chart templates: - _helpers.tpl: new `cube.clusterDomain` helper honoring global.clusterDomain > cubeNode.dns.clusterDomain > cluster.local, and `cube.apiBindPort` derived from controlPlane.api.bind. All in- cluster Service FQDNs (cube.masterEndpoint / cube.apiEndpoint / cube.mysqlHost / cube.redisHost) now build the DNS suffix through the helper so clusters with a non-default kubelet cluster-domain resolve correctly. - api.yaml: containerPort is no longer hardcoded to 3000; it is derived from controlPlane.api.bind via cube.apiBindPort. - webui.yaml + tests/node-health.yaml: replace remaining `svc.cluster.local` literals with cube.clusterDomain. - diagnostics.yaml: rollout status commands use cube.nodeName / cube.proxyName / cube.mysqlName / cube.redisName helpers instead of `${RELEASE}-node` string concatenation, so the diagnostic script keeps working when nameOverride / fullnameOverride is set. - dns.yaml: add `errors` plugin, gate `log` behind cubeDns.logQueries (default off), align cache TTL with the 60s TTL emitted by the template plugin (previously 300s, hiding IP changes for up to five minutes). - validate.yaml: reject the CHANGE_ME_* password sentinels when the built-in MySQL / Redis is enabled so a chart install cannot succeed with the trivially guessable defaults. - node-daemonset.yaml: each cube-node probe (startup / readiness / liveness) now supports an `exec:` override in addition to the TCP socket check so operators can detect deadlocked processes that keep port 9999 open. Default resource requests/limits are shipped for the cube-node container so the DaemonSet does not enter BestEffort QoS on a busy compute node. - proxy-node.yaml: the 50-line inline shell that generated global.conf is extracted into images/scripts/cube-proxy-node-entrypoint.sh, delivered by the chart as a ConfigMap and mounted into the Pod at /etc/cube-proxy-node. The script can now be linted / tested / diff- reviewed as a regular source file. - tests/node-health.yaml: CubeEgress test extracts container names from the pods JSON before matching so cube-egress does not match labels or annotations that happen to contain that string; the CubeProxy test now issues an actual readiness curl on the in- cluster Service endpoint instead of only printing a message; the service-mode DNS test gains a compute placement fallback and uses cube.clusterDomain everywhere; the node-image-test no longer pulls the multi-GB cube-node runtime image just to run `test -f`, it now reuses the live DaemonSet Pod via a curl-based readiness check (gated by helmTest.nodeImage.enabled). values.yaml: - global.clusterDomain: "" — new knob; empty falls back to cubeNode.dns.clusterDomain and then "cluster.local". - mysql.password / mysql.rootPassword / redis.password: switch the in-repo defaults from cube_pass / cube_root / ceuhvu123 to CHANGE_ME_* sentinels; validate.yaml refuses to render with them. - bootstrap.pvmHostKernel.bootArgs: keeps the `nopti pti=off` default (kvm_pvm requires KPTI off) but adds a security trade-off explanation so operators consciously accept it. - cubeNode.resources: sensible baseline requests/limits so a busy cube-node is never evicted before the sidecars. - cubeNode.probes.*.exec: [] — operator-supplied application-level probe overrides that override the TCP socket check. - cubeDns.cacheTTL / cubeDns.logQueries: matched to the 60s A-record TTL and default-off log plugin respectively. - cubeEgress.resources / cubeEgress.netResources: default requests/ limits for the two egress sidecars. Image scripts / Dockerfiles: - images/build-cube-images.sh: `download_file` now accepts an optional expected SHA256; three new env vars ONE_CLICK_SHA256, PVM_KERNEL_RPM_SHA256, PVM_KERNEL_DEB_SHA256 activate the check. The `--retry-all-errors` curl flag moves to opt-in via CURL_RETRY_ALL_ERRORS=1 so a 404 no longer costs 9 retries. Adds a `need sha256sum` guard. - images/scripts/pvm-host-bootstrap.sh: introduce `host_run` for argv-style execution without `sh -c`, and shell-quote DESIRED_KERNEL_PATTERN before splicing it into the host shell to close the command injection vector reported by the review. The post-reboot `sleep 3600` shrinks to `${REBOOT_WAIT_SECONDS:-120}` so an init container failure surfaces to the DaemonSet in minutes instead of an hour. - images/scripts/cube-node-entrypoint.sh: escape sed replacement input via `sed_escape_replacement` and reject non-integer values for CUBE_TAP_INIT_NUM / CUBE_CGROUP_POOL_SIZE / CUBE_WORKFLOW_ CONCURRENT so operator-supplied env vars cannot inject configuration content. The hardcoded cube-master.cube-system endpoint fallback is replaced with a fail-fast check because a wrong-namespace default silently targets the wrong control plane. - images/scripts/cube-node-init.sh: document the security model of `host_chroot_sh` / `host_mount_sh` in a block comment so operators and reviewers understand that these helpers are a deliberate, architecturally required container escape and treat the images accordingly. - images/{cube-node,cube-node-init,cubemastercli,cube-egress-net, cube-pvm-host-bootstrap}/Dockerfile: bump the ubuntu base from 20.04 (EOL April 2025) to 22.04. - images/scripts/cube-proxy-node-entrypoint.sh: new file; the extracted proxy entrypoint referenced above. - chart/files/cube-proxy-node/: mirror copy consumed by the ConfigMap template so `helm template` can inline the script content. Verified: - helm lint passes. - helm template with representative overrides renders 34 resources. - helm template with global.clusterDomain=example.internal makes every in-cluster Service FQDN carry the override suffix; the default render keeps `cluster.local`. - helm template with controlPlane.api.bind=0.0.0.0:8888 changes the cube-api Pod containerPort to 8888 while the Service port stays configurable via controlPlane.api.service.port. - helm template with default mysql/redis passwords fails validate.yaml with an actionable CHANGE_ME_* message. - bash -n on the modified scripts passes.
2 participants
Add a Helm chart under deploy/kubernetes/ so CubeSandbox v0.5.0 can be delivered on Kubernetes/TKE from an internal registry.
deploy/kubernetes/chart/
deploy/kubernetes/images/
Default registry / tag:
Verified: