Skip to content

Commit 9b75c6c

Browse files
docs(v1.7): describe the Proxmox VE substrate for tenant clusters
v1.7.0 ships the Proxmox VE substrate for tenant Kubernetes workers and its release notes lead with it, yet the v1.7 docs have no page for it: #761 adds one to next/ only. The page is #761's with the rewrite release_next.sh applies, checked against the v1.7.0 tag. Four things did not hold there and are corrected in this copy: - KubernetesNodes has no replicas field; a pool is sized by minReplicas and maxReplicas, which the tag's own Proxmox e2e manifest sets. - The CCM and CSI controller Deployments are <release>-pccm and <release>-pcsi-controller, and a Kubernetes release is named kubernetes-<name>, so the cluster px runs kubernetes-px-pccm. The verify command and the troubleshooting row now name those. - bundles.enabledPackages is read only from the platform values on the cozystack.cozystack-platform Package; no cozystack ConfigMap feeds it. - The CSI HelmRelease depends on the tenant's Cilium release, not on the CCM, which is a plain Deployment, so the troubleshooting fix only says to wait for the CCM. next/ stays with #761. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
1 parent 5fc1e18 commit 9b75c6c

1 file changed

Lines changed: 334 additions & 0 deletions

File tree

Lines changed: 334 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,334 @@
1+
---
2+
title: "Running Tenant Kubernetes Workers on Proxmox VE"
3+
linkTitle: "Proxmox VE Substrate"
4+
description: "Run the worker nodes of managed Kubernetes clusters on an external Proxmox VE cluster instead of KubeVirt, using the Cluster API Proxmox provider, the Proxmox cloud-controller-manager and the Proxmox CSI driver."
5+
weight: 70
6+
---
7+
8+
{{< note >}}
9+
The Proxmox VE substrate is **opt-in and experimental** as of Cozystack v1.7.
10+
It is shipped in the platform, tested by its maintainer on Proxmox VE 9.2, and
11+
not yet covered by the project's release gate. Track the decision on support
12+
tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481).
13+
{{< /note >}}
14+
15+
## What it is
16+
17+
By default a managed (tenant) Kubernetes cluster in Cozystack has its control
18+
plane run by Kamaji in the management cluster and its worker nodes run as
19+
KubeVirt virtual machines inside the same cluster.
20+
21+
With `substrate: proxmox` the control plane stays exactly where it is, and the
22+
**workers are cloned as VMs on an external Proxmox VE cluster** through the
23+
Cluster API provider for Proxmox (capmox). No nested virtualization, no
24+
KubeVirt on the worker path.
25+
26+
```
27+
Management cluster (Cozystack) Proxmox VE cluster
28+
┌──────────────────────────────────┐ ┌─────────────────────────┐
29+
│ Kamaji control plane (tenant A) │◄── join ──│ worker VM (pool: a) │
30+
│ capmox + in-cluster IPAM │── clone ─►│ worker VM (pool: a) │
31+
│ proxmox-ccm (tenant ns) │── API ───►│ worker VM (pool: b) │
32+
│ proxmox-csi controller (tenant ns)│── API ──►│ Talos VM template │
33+
└──────────────────────────────────┘ └─────────────────────────┘
34+
```
35+
36+
| Component | Where it runs | What it does |
37+
|---|---|---|
38+
| capmox (`ProxmoxCluster`, `ProxmoxMachineTemplate`) | management cluster, `cozy-cluster-api` | clones worker VMs from a Talos template, assigns static IPs from an in-cluster IPAM pool |
39+
| Proxmox cloud-controller-manager | management cluster, tenant namespace | sets `providerID`, region/zone labels, clears the `uninitialized` taint on tenant Nodes over the tenant's admin kubeconfig |
40+
| Proxmox CSI controller | management cluster, tenant namespace | creates and attaches Proxmox disks for tenant PersistentVolumes |
41+
| Proxmox CSI node plugin | tenant cluster | mounts the disks; carries no hypervisor credential |
42+
43+
The hypervisor token never enters the tenant cluster.
44+
45+
## Support matrix
46+
47+
| | Verified | Notes |
48+
|---|---|---|
49+
| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 |
50+
| Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x |
51+
| capmox | v0.7.7 (shipped) | |
52+
| Talos | the version in `kubernetes-nodes` `talos.version` | template must match it |
53+
| Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested |
54+
| Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* |
55+
| Networking | VM NIC on a Proxmox bridge, optional VLAN tag | workers must reach the tenant apiserver's LoadBalancer IP |
56+
57+
## Prerequisites
58+
59+
On the **Proxmox VE** side:
60+
61+
- an API endpoint `https://<pve-host>:8006` reachable from the management cluster;
62+
- a bridge the workers attach to, with L3 reachability to the management
63+
cluster's LoadBalancer range (the tenant apiserver is published as a
64+
`LoadBalancer` Service on this substrate);
65+
- a range of free static IPv4 addresses on that L2 for the workers, not
66+
overlapping any DHCP range;
67+
- a storage the Talos template lives on (workers are linked clones of it by default);
68+
- reachable DNS resolvers for the workers (they cannot use the management
69+
cluster's CoreDNS).
70+
71+
On the **Cozystack** side:
72+
73+
- a working management cluster with the `iaas` bundle;
74+
- a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address.
75+
76+
## 1. Enable the packages
77+
78+
Add the Proxmox infrastructure provider to the platform's enabled packages.
79+
The in-cluster IPAM provider is pulled in as its dependency.
80+
81+
```yaml
82+
# spec.components.platform.values of the cozystack.cozystack-platform Package
83+
bundles:
84+
enabledPackages:
85+
- cozystack.capi-provider-infra-proxmox
86+
```
87+
88+
Before the provider is deployed, create the Secret it authenticates with.
89+
capi-operator fetches it first; while it is missing nothing is installed and
90+
the `InfrastructureProvider` reports the fetch error.
91+
92+
```bash
93+
kubectl -n cozy-cluster-api create secret generic capmox-config \
94+
--from-literal=PROXMOX_URL='https://pve.example.com:8006' \
95+
--from-literal=PROXMOX_TOKEN='capmox@pve!cozystack' \
96+
--from-literal=PROXMOX_SECRET='<token secret>'
97+
```
98+
99+
Three things that are easy to get wrong:
100+
101+
- `PROXMOX_URL` is the bare `https://host:8006`; capmox appends `/api2/json` itself.
102+
- `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone.
103+
- a rotated token is picked up only after a restart of the capmox manager pod.
104+
105+
Check:
106+
107+
```bash
108+
kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders
109+
kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam'
110+
```
111+
112+
## 2. Prepare Proxmox: user, tokens, pool
113+
114+
Create a user and tokens for the platform. Use **privilege-separated tokens**
115+
and grant the privileges to the user as well: Proxmox caps a token at its
116+
user's permissions.
117+
118+
For capmox itself, follow the provider's own
119+
[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md):
120+
121+
```bash
122+
pveum user add capmox@pve
123+
pveum aclmod / -user capmox@pve -role PVEVMAdmin
124+
pveum user token add capmox@pve cozystack -privsep 0
125+
```
126+
127+
`PVEVMAdmin` on `/` is what the provider documents; a narrower set is
128+
described in its advanced-setup docs. This token is platform-wide: capmox
129+
clones the workers of every tenant with it (see *Isolation*).
130+
131+
For each **tenant** create a resource pool and a dedicated storage, and grant
132+
the tenant's CCM and CSI tokens their privileges on those paths only:
133+
134+
| Component | Privileges | Path |
135+
|---|---|---|
136+
| CSI controller | `VM.Audit`, `VM.Config.Disk` | `/pool/<tenant-pool>` |
137+
| CSI controller | `Datastore.Allocate`, `Datastore.AllocateSpace`, `Datastore.Audit` | `/storage/<tenant-storage>` |
138+
| CCM | `VM.Audit`, `VM.GuestAgent.Audit` | `/pool/<tenant-pool>` |
139+
| CCM | `Sys.Audit` | `/` |
140+
141+
```bash
142+
pveum pool add tenant-a
143+
pveum user add tenant-a@pve
144+
pveum user token add tenant-a@pve csi --privsep 1
145+
pveum user token add tenant-a@pve ccm --privsep 1
146+
pveum role add CozyCSI -privs "VM.Audit VM.Config.Disk"
147+
pveum role add CozyCSIStorage -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.Audit"
148+
pveum role add CozyCCM -privs "VM.Audit VM.GuestAgent.Audit"
149+
# the user ...
150+
pveum acl modify /pool/tenant-a --users tenant-a@pve --roles CozyCSI,CozyCCM
151+
pveum acl modify /storage/tenant-a-disks --users tenant-a@pve --roles CozyCSIStorage
152+
pveum acl modify / --users tenant-a@pve --roles PVEAuditor
153+
# ... and each token, which Proxmox caps at the user's permissions
154+
pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!csi' --roles CozyCSI
155+
pveum acl modify /storage/tenant-a-disks --tokens 'tenant-a@pve!csi' --roles CozyCSIStorage
156+
pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!ccm' --roles CozyCCM
157+
pveum acl modify / --tokens 'tenant-a@pve!ccm' --roles PVEAuditor
158+
pveum user token permissions tenant-a@pve csi --path /pool/tenant-a
159+
```
160+
161+
Why a pool: capmox adds every worker VM it clones to `proxmox.pool`,
162+
replacements included, so an ACL on `/pool/<name>` follows the tenant's VMs
163+
without being granted again.
164+
165+
## 3. Build the Talos VM template
166+
167+
Workers are cloned from a Talos VM template that already exists on the
168+
hypervisor. Build it with the script shipped in the repository; it reads
169+
`talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so
170+
the template matches what the chart will render into the machine config.
171+
172+
```bash
173+
# prints the program it would run, for review
174+
hack/proxmox-talos-template.sh print 9000 local-zfs
175+
176+
# creates the template over ssh on the PVE node
177+
COZY_PVE_SSH=root@pve.example.com hack/proxmox-talos-template.sh create 9000 local-zfs
178+
```
179+
180+
The script prints the tags to put into `proxmox.templateTags`. Do **not**
181+
build the template from the platform's `nocloud-amd64.raw.xz`: that is the
182+
bare-metal node image, and its `zfs` extension keeps a VM from completing its
183+
boot, rebooting it every ~70 minutes.
184+
185+
## 4. Create the tenant cluster
186+
187+
```yaml
188+
apiVersion: apps.cozystack.io/v1alpha1
189+
kind: Kubernetes
190+
metadata:
191+
name: px
192+
namespace: tenant-a
193+
spec:
194+
substrate: proxmox
195+
proxmox:
196+
allowedNodes: ["pve1", "pve2"] # empty = every node
197+
dnsServers: ["10.0.0.1"]
198+
insecure: true # stock PVE serves a self-signed cert
199+
ipv4Config:
200+
addresses: ["10.0.0.120-10.0.0.170"]
201+
gateway: "10.0.0.1"
202+
prefix: 24
203+
ccm:
204+
credentialsSecretName: proxmox-ccm # keys: url, token_id, token_secret, region
205+
csi:
206+
enabled: true
207+
credentialsSecretName: proxmox-csi
208+
storageClasses:
209+
- name: proxmox
210+
storage: tenant-a-disks
211+
fstype: ext4
212+
reclaimPolicy: Delete
213+
# ... the rest of the usual Kubernetes spec
214+
```
215+
216+
The two credential Secrets live in the **tenant's namespace of the management
217+
cluster** with the keys `url`, `token_id`, `token_secret`, `region`
218+
(`region` is the Proxmox cluster's name; the controllers put it on the
219+
Nodes as `topology.kubernetes.io/region`. Note the URL here carries
220+
`/api2/json`, unlike capmox's `PROXMOX_URL`):
221+
222+
```bash
223+
kubectl -n tenant-a create secret generic proxmox-csi \
224+
--from-literal=url='https://pve.example.com:8006/api2/json' \
225+
--from-literal=token_id='tenant-a@pve!csi' \
226+
--from-literal=token_secret='<secret>' \
227+
--from-literal=region='pve'
228+
```
229+
230+
`substrate` is immutable: switching an existing cluster is refused. Create a
231+
new one.
232+
233+
## 5. Add a worker pool
234+
235+
```yaml
236+
apiVersion: apps.cozystack.io/v1alpha1
237+
kind: KubernetesNodes
238+
metadata:
239+
name: px-md0
240+
namespace: tenant-a
241+
spec:
242+
cluster: px
243+
substrate: proxmox
244+
minReplicas: 2
245+
maxReplicas: 2
246+
resources:
247+
cpu: 4
248+
memory: 8Gi
249+
proxmox:
250+
templateTags: ["cozystack", "talos", "talos-v1.13.6", "schematic-ce4c9805"]
251+
full: false # linked clone (default)
252+
network:
253+
bridge: vmbr0
254+
vlan: 60 # optional
255+
pool: tenant-a
256+
dnsServers: ["10.0.0.1"]
257+
```
258+
259+
`templateTags` must be **equal** to the template's tag set, not a subset: a
260+
missing tag yields `found 0 VM templates with tags ...` even though the tags
261+
are plainly present on the template.
262+
263+
`resources.cpu` / `memory` are converted to capmox's `numCores` / `memoryMiB`
264+
(rounded up).
265+
266+
## 6. Verify
267+
268+
```bash
269+
# management cluster
270+
kubectl -n tenant-a get cluster,proxmoxcluster,machinedeployment,machine
271+
kubectl -n tenant-a get proxmoxmachines -o custom-columns='NAME:.metadata.name,VMID:.spec.virtualMachineID,NODE:.status.proxmoxNode,IP:.status.ipAddresses.net0.ipv4,STATE:.status.vmStatus'
272+
kubectl -n tenant-a get inclusterippools,ipaddresses.ipam.cluster.x-k8s.io
273+
kubectl -n tenant-a get deploy kubernetes-px-pccm kubernetes-px-pcsi-controller
274+
275+
# tenant cluster
276+
kubectl --kubeconfig <tenant kubeconfig> get nodes -o wide
277+
kubectl --kubeconfig <tenant kubeconfig> get nodes -o custom-columns='NODE:.metadata.name,PROVIDER-ID:.spec.providerID,TAINTS:.spec.taints[*].key'
278+
kubectl --kubeconfig <tenant kubeconfig> -n csi-proxmox get pods
279+
```
280+
281+
A healthy node has the same `providerID` as its CAPI `Machine`
282+
(`proxmox://<SMBIOS UUID>`; the CCM runs in its `capmox` mode for that), the
283+
`topology.kubernetes.io/region|zone` labels, and no
284+
`node.cloudprovider.kubernetes.io/uninitialized` taint.
285+
286+
What is covered by the integration and was verified live on PVE 9.2:
287+
PVC create → attach → online resize → delete; Cluster Autoscaler scale-up and
288+
scale-down through the pool; rolling replacement of a worker;
289+
MachineHealthCheck.
290+
291+
## Isolation: what the design gives and what it does not
292+
293+
- The hypervisor token is composed into the CCM and CSI controllers in the
294+
management cluster; nothing copies it into the tenant cluster.
295+
- The tenant is cluster-admin of its own cluster, and the controllers act on
296+
what its objects say: the CSI controller attaches the disk a PV names to the
297+
VM a Node names. **Scope comes from the Proxmox ACL**, not from the
298+
controllers: one pool and one storage per tenant.
299+
- A storage shared between tenants cannot be scoped: `Datastore.Allocate` on
300+
it reaches every volume on it.
301+
- capmox itself (VM creation) currently uses one platform-wide token for all
302+
tenants; per-tenant `credentialsRef` on `ProxmoxCluster` is being worked on
303+
upstream.
304+
305+
## Limitations (v1.7)
306+
307+
- Opt-in, experimental; no release gate yet (an e2e lane exists upstream, but its nightly run is opt-in and currently skipped).
308+
- Workers only. Cozystack's `vm-disk` / `vm-instance` applications still run on KubeVirt; a Proxmox substrate for them is in design.
309+
- Tenant `Kubernetes` kinds are published as part of the KubeVirt variant, so a management cluster still needs the KubeVirt packages installed even if every pool runs on Proxmox.
310+
- No DHCP: capmox assigns static addresses from `ipv4Config` only.
311+
- Linked clones keep the template's base snapshot referenced; rotate templates by creating a new VMID with new tags and rolling the pool.
312+
- No live migration awareness: the CCM reports a VM it cannot see as gone and deletes the Node.
313+
- Cozystack's Terraform provider does not model `substrate`/`proxmox.*` yet; `terraform apply` on such a cluster overwrites them.
314+
315+
## Troubleshooting
316+
317+
| Symptom | Cause | Fix |
318+
|---|---|---|
319+
| `InfrastructureProvider` not installing, fetch error | `capmox-config` Secret missing or in the wrong namespace | create it in `cozy-cluster-api` |
320+
| every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` |
321+
| `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly |
322+
| Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS |
323+
| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n <tenant ns> logs deploy/kubernetes-<cluster>-pccm` |
324+
| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM |
325+
| VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` |
326+
| `TalosConfigTemplate.Spec is immutable` | apiserver address changed on an existing cluster | delete the template; the reconcile Job recreates it |
327+
| capmox stuck at `Waiting for Cluster Controller to set OwnerRef` | `cluster.x-k8s.io/managed-by` annotation on the `ProxmoxCluster` | not expected from the chart; report it |
328+
329+
## See also
330+
331+
- [Managed Kubernetes]({{% ref "/docs/v1.7/kubernetes" %}}) — the application this page extends
332+
- Cluster API provider for Proxmox: https://github.com/ionos-cloud/cluster-api-provider-proxmox
333+
- Proxmox CCM / CSI: https://github.com/sergelogvinov/proxmox-cloud-controller-manager, https://github.com/sergelogvinov/proxmox-csi-plugin
334+
- Support-tier proposal: https://github.com/cozystack/community/pull/98

0 commit comments

Comments
 (0)