|
| 1 | +--- |
| 2 | +title: "Running Tenant Kubernetes Workers on Proxmox VE" |
| 3 | +linkTitle: "Proxmox VE Substrate" |
| 4 | +description: "Run the worker nodes of managed Kubernetes clusters on an external Proxmox VE cluster instead of KubeVirt, using the Cluster API Proxmox provider, the Proxmox cloud-controller-manager and the Proxmox CSI driver." |
| 5 | +weight: 70 |
| 6 | +--- |
| 7 | + |
| 8 | +{{< note >}} |
| 9 | +The Proxmox VE substrate is **opt-in and experimental** as of Cozystack v1.7. |
| 10 | +It is shipped in the platform, tested by its maintainer on Proxmox VE 9.2, and |
| 11 | +not yet covered by the project's release gate. Track the decision on support |
| 12 | +tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481). |
| 13 | +{{< /note >}} |
| 14 | + |
| 15 | +## What it is |
| 16 | + |
| 17 | +By default a managed (tenant) Kubernetes cluster in Cozystack has its control |
| 18 | +plane run by Kamaji in the management cluster and its worker nodes run as |
| 19 | +KubeVirt virtual machines inside the same cluster. |
| 20 | + |
| 21 | +With `substrate: proxmox` the control plane stays exactly where it is, and the |
| 22 | +**workers are cloned as VMs on an external Proxmox VE cluster** through the |
| 23 | +Cluster API provider for Proxmox (capmox). No nested virtualization, no |
| 24 | +KubeVirt on the worker path. |
| 25 | + |
| 26 | +``` |
| 27 | +Management cluster (Cozystack) Proxmox VE cluster |
| 28 | +┌──────────────────────────────────┐ ┌─────────────────────────┐ |
| 29 | +│ Kamaji control plane (tenant A) │◄── join ──│ worker VM (pool: a) │ |
| 30 | +│ capmox + in-cluster IPAM │── clone ─►│ worker VM (pool: a) │ |
| 31 | +│ proxmox-ccm (tenant ns) │── API ───►│ worker VM (pool: b) │ |
| 32 | +│ proxmox-csi controller (tenant ns)│── API ──►│ Talos VM template │ |
| 33 | +└──────────────────────────────────┘ └─────────────────────────┘ |
| 34 | +``` |
| 35 | + |
| 36 | +| Component | Where it runs | What it does | |
| 37 | +|---|---|---| |
| 38 | +| capmox (`ProxmoxCluster`, `ProxmoxMachineTemplate`) | management cluster, `cozy-cluster-api` | clones worker VMs from a Talos template, assigns static IPs from an in-cluster IPAM pool | |
| 39 | +| Proxmox cloud-controller-manager | management cluster, tenant namespace | sets `providerID`, region/zone labels, clears the `uninitialized` taint on tenant Nodes over the tenant's admin kubeconfig | |
| 40 | +| Proxmox CSI controller | management cluster, tenant namespace | creates and attaches Proxmox disks for tenant PersistentVolumes | |
| 41 | +| Proxmox CSI node plugin | tenant cluster | mounts the disks; carries no hypervisor credential | |
| 42 | + |
| 43 | +The hypervisor token never enters the tenant cluster. |
| 44 | + |
| 45 | +## Support matrix |
| 46 | + |
| 47 | +| | Verified | Notes | |
| 48 | +|---|---|---| |
| 49 | +| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 | |
| 50 | +| Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x | |
| 51 | +| capmox | v0.7.7 (shipped) | | |
| 52 | +| Talos | the version in `kubernetes-nodes` `talos.version` | template must match it | |
| 53 | +| Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested | |
| 54 | +| Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* | |
| 55 | +| Networking | VM NIC on a Proxmox bridge, optional VLAN tag | workers must reach the tenant apiserver's LoadBalancer IP | |
| 56 | + |
| 57 | +## Prerequisites |
| 58 | + |
| 59 | +On the **Proxmox VE** side: |
| 60 | + |
| 61 | +- an API endpoint `https://<pve-host>:8006` reachable from the management cluster; |
| 62 | +- a bridge the workers attach to, with L3 reachability to the management |
| 63 | + cluster's LoadBalancer range (the tenant apiserver is published as a |
| 64 | + `LoadBalancer` Service on this substrate); |
| 65 | +- a range of free static IPv4 addresses on that L2 for the workers, not |
| 66 | + overlapping any DHCP range; |
| 67 | +- a storage the Talos template lives on (workers are linked clones of it by default); |
| 68 | +- reachable DNS resolvers for the workers (they cannot use the management |
| 69 | + cluster's CoreDNS). |
| 70 | + |
| 71 | +On the **Cozystack** side: |
| 72 | + |
| 73 | +- a working management cluster with the `iaas` bundle; |
| 74 | +- a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address. |
| 75 | + |
| 76 | +## 1. Enable the packages |
| 77 | + |
| 78 | +Add the Proxmox infrastructure provider to the platform's enabled packages. |
| 79 | +The in-cluster IPAM provider is pulled in as its dependency. |
| 80 | + |
| 81 | +```yaml |
| 82 | +# spec.components.platform.values of the cozystack.cozystack-platform Package |
| 83 | +bundles: |
| 84 | + enabledPackages: |
| 85 | + - cozystack.capi-provider-infra-proxmox |
| 86 | +``` |
| 87 | +
|
| 88 | +Before the provider is deployed, create the Secret it authenticates with. |
| 89 | +capi-operator fetches it first; while it is missing nothing is installed and |
| 90 | +the `InfrastructureProvider` reports the fetch error. |
| 91 | + |
| 92 | +```bash |
| 93 | +kubectl -n cozy-cluster-api create secret generic capmox-config \ |
| 94 | + --from-literal=PROXMOX_URL='https://pve.example.com:8006' \ |
| 95 | + --from-literal=PROXMOX_TOKEN='capmox@pve!cozystack' \ |
| 96 | + --from-literal=PROXMOX_SECRET='<token secret>' |
| 97 | +``` |
| 98 | + |
| 99 | +Three things that are easy to get wrong: |
| 100 | + |
| 101 | +- `PROXMOX_URL` is the bare `https://host:8006`; capmox appends `/api2/json` itself. |
| 102 | +- `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone. |
| 103 | +- a rotated token is picked up only after a restart of the capmox manager pod. |
| 104 | + |
| 105 | +Check: |
| 106 | + |
| 107 | +```bash |
| 108 | +kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders |
| 109 | +kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam' |
| 110 | +``` |
| 111 | + |
| 112 | +## 2. Prepare Proxmox: user, tokens, pool |
| 113 | + |
| 114 | +Create a user and tokens for the platform. Use **privilege-separated tokens** |
| 115 | +and grant the privileges to the user as well: Proxmox caps a token at its |
| 116 | +user's permissions. |
| 117 | + |
| 118 | +For capmox itself, follow the provider's own |
| 119 | +[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md): |
| 120 | + |
| 121 | +```bash |
| 122 | +pveum user add capmox@pve |
| 123 | +pveum aclmod / -user capmox@pve -role PVEVMAdmin |
| 124 | +pveum user token add capmox@pve cozystack -privsep 0 |
| 125 | +``` |
| 126 | + |
| 127 | +`PVEVMAdmin` on `/` is what the provider documents; a narrower set is |
| 128 | +described in its advanced-setup docs. This token is platform-wide: capmox |
| 129 | +clones the workers of every tenant with it (see *Isolation*). |
| 130 | + |
| 131 | +For each **tenant** create a resource pool and a dedicated storage, and grant |
| 132 | +the tenant's CCM and CSI tokens their privileges on those paths only: |
| 133 | + |
| 134 | +| Component | Privileges | Path | |
| 135 | +|---|---|---| |
| 136 | +| CSI controller | `VM.Audit`, `VM.Config.Disk` | `/pool/<tenant-pool>` | |
| 137 | +| CSI controller | `Datastore.Allocate`, `Datastore.AllocateSpace`, `Datastore.Audit` | `/storage/<tenant-storage>` | |
| 138 | +| CCM | `VM.Audit`, `VM.GuestAgent.Audit` | `/pool/<tenant-pool>` | |
| 139 | +| CCM | `Sys.Audit` | `/` | |
| 140 | + |
| 141 | +```bash |
| 142 | +pveum pool add tenant-a |
| 143 | +pveum user add tenant-a@pve |
| 144 | +pveum user token add tenant-a@pve csi --privsep 1 |
| 145 | +pveum user token add tenant-a@pve ccm --privsep 1 |
| 146 | +pveum role add CozyCSI -privs "VM.Audit VM.Config.Disk" |
| 147 | +pveum role add CozyCSIStorage -privs "Datastore.Allocate Datastore.AllocateSpace Datastore.Audit" |
| 148 | +pveum role add CozyCCM -privs "VM.Audit VM.GuestAgent.Audit" |
| 149 | +# the user ... |
| 150 | +pveum acl modify /pool/tenant-a --users tenant-a@pve --roles CozyCSI,CozyCCM |
| 151 | +pveum acl modify /storage/tenant-a-disks --users tenant-a@pve --roles CozyCSIStorage |
| 152 | +pveum acl modify / --users tenant-a@pve --roles PVEAuditor |
| 153 | +# ... and each token, which Proxmox caps at the user's permissions |
| 154 | +pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!csi' --roles CozyCSI |
| 155 | +pveum acl modify /storage/tenant-a-disks --tokens 'tenant-a@pve!csi' --roles CozyCSIStorage |
| 156 | +pveum acl modify /pool/tenant-a --tokens 'tenant-a@pve!ccm' --roles CozyCCM |
| 157 | +pveum acl modify / --tokens 'tenant-a@pve!ccm' --roles PVEAuditor |
| 158 | +pveum user token permissions tenant-a@pve csi --path /pool/tenant-a |
| 159 | +``` |
| 160 | + |
| 161 | +Why a pool: capmox adds every worker VM it clones to `proxmox.pool`, |
| 162 | +replacements included, so an ACL on `/pool/<name>` follows the tenant's VMs |
| 163 | +without being granted again. |
| 164 | + |
| 165 | +## 3. Build the Talos VM template |
| 166 | + |
| 167 | +Workers are cloned from a Talos VM template that already exists on the |
| 168 | +hypervisor. Build it with the script shipped in the repository; it reads |
| 169 | +`talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so |
| 170 | +the template matches what the chart will render into the machine config. |
| 171 | + |
| 172 | +```bash |
| 173 | +# prints the program it would run, for review |
| 174 | +hack/proxmox-talos-template.sh print 9000 local-zfs |
| 175 | +
|
| 176 | +# creates the template over ssh on the PVE node |
| 177 | +COZY_PVE_SSH=root@pve.example.com hack/proxmox-talos-template.sh create 9000 local-zfs |
| 178 | +``` |
| 179 | + |
| 180 | +The script prints the tags to put into `proxmox.templateTags`. Do **not** |
| 181 | +build the template from the platform's `nocloud-amd64.raw.xz`: that is the |
| 182 | +bare-metal node image, and its `zfs` extension keeps a VM from completing its |
| 183 | +boot, rebooting it every ~70 minutes. |
| 184 | + |
| 185 | +## 4. Create the tenant cluster |
| 186 | + |
| 187 | +```yaml |
| 188 | +apiVersion: apps.cozystack.io/v1alpha1 |
| 189 | +kind: Kubernetes |
| 190 | +metadata: |
| 191 | + name: px |
| 192 | + namespace: tenant-a |
| 193 | +spec: |
| 194 | + substrate: proxmox |
| 195 | + proxmox: |
| 196 | + allowedNodes: ["pve1", "pve2"] # empty = every node |
| 197 | + dnsServers: ["10.0.0.1"] |
| 198 | + insecure: true # stock PVE serves a self-signed cert |
| 199 | + ipv4Config: |
| 200 | + addresses: ["10.0.0.120-10.0.0.170"] |
| 201 | + gateway: "10.0.0.1" |
| 202 | + prefix: 24 |
| 203 | + ccm: |
| 204 | + credentialsSecretName: proxmox-ccm # keys: url, token_id, token_secret, region |
| 205 | + csi: |
| 206 | + enabled: true |
| 207 | + credentialsSecretName: proxmox-csi |
| 208 | + storageClasses: |
| 209 | + - name: proxmox |
| 210 | + storage: tenant-a-disks |
| 211 | + fstype: ext4 |
| 212 | + reclaimPolicy: Delete |
| 213 | + # ... the rest of the usual Kubernetes spec |
| 214 | +``` |
| 215 | + |
| 216 | +The two credential Secrets live in the **tenant's namespace of the management |
| 217 | +cluster** with the keys `url`, `token_id`, `token_secret`, `region` |
| 218 | +(`region` is the Proxmox cluster's name; the controllers put it on the |
| 219 | +Nodes as `topology.kubernetes.io/region`. Note the URL here carries |
| 220 | +`/api2/json`, unlike capmox's `PROXMOX_URL`): |
| 221 | + |
| 222 | +```bash |
| 223 | +kubectl -n tenant-a create secret generic proxmox-csi \ |
| 224 | + --from-literal=url='https://pve.example.com:8006/api2/json' \ |
| 225 | + --from-literal=token_id='tenant-a@pve!csi' \ |
| 226 | + --from-literal=token_secret='<secret>' \ |
| 227 | + --from-literal=region='pve' |
| 228 | +``` |
| 229 | + |
| 230 | +`substrate` is immutable: switching an existing cluster is refused. Create a |
| 231 | +new one. |
| 232 | + |
| 233 | +## 5. Add a worker pool |
| 234 | + |
| 235 | +```yaml |
| 236 | +apiVersion: apps.cozystack.io/v1alpha1 |
| 237 | +kind: KubernetesNodes |
| 238 | +metadata: |
| 239 | + name: px-md0 |
| 240 | + namespace: tenant-a |
| 241 | +spec: |
| 242 | + cluster: px |
| 243 | + substrate: proxmox |
| 244 | + minReplicas: 2 |
| 245 | + maxReplicas: 2 |
| 246 | + resources: |
| 247 | + cpu: 4 |
| 248 | + memory: 8Gi |
| 249 | + proxmox: |
| 250 | + templateTags: ["cozystack", "talos", "talos-v1.13.6", "schematic-ce4c9805"] |
| 251 | + full: false # linked clone (default) |
| 252 | + network: |
| 253 | + bridge: vmbr0 |
| 254 | + vlan: 60 # optional |
| 255 | + pool: tenant-a |
| 256 | + dnsServers: ["10.0.0.1"] |
| 257 | +``` |
| 258 | + |
| 259 | +`templateTags` must be **equal** to the template's tag set, not a subset: a |
| 260 | +missing tag yields `found 0 VM templates with tags ...` even though the tags |
| 261 | +are plainly present on the template. |
| 262 | + |
| 263 | +`resources.cpu` / `memory` are converted to capmox's `numCores` / `memoryMiB` |
| 264 | +(rounded up). |
| 265 | + |
| 266 | +## 6. Verify |
| 267 | + |
| 268 | +```bash |
| 269 | +# management cluster |
| 270 | +kubectl -n tenant-a get cluster,proxmoxcluster,machinedeployment,machine |
| 271 | +kubectl -n tenant-a get proxmoxmachines -o custom-columns='NAME:.metadata.name,VMID:.spec.virtualMachineID,NODE:.status.proxmoxNode,IP:.status.ipAddresses.net0.ipv4,STATE:.status.vmStatus' |
| 272 | +kubectl -n tenant-a get inclusterippools,ipaddresses.ipam.cluster.x-k8s.io |
| 273 | +kubectl -n tenant-a get deploy kubernetes-px-pccm kubernetes-px-pcsi-controller |
| 274 | +
|
| 275 | +# tenant cluster |
| 276 | +kubectl --kubeconfig <tenant kubeconfig> get nodes -o wide |
| 277 | +kubectl --kubeconfig <tenant kubeconfig> get nodes -o custom-columns='NODE:.metadata.name,PROVIDER-ID:.spec.providerID,TAINTS:.spec.taints[*].key' |
| 278 | +kubectl --kubeconfig <tenant kubeconfig> -n csi-proxmox get pods |
| 279 | +``` |
| 280 | + |
| 281 | +A healthy node has the same `providerID` as its CAPI `Machine` |
| 282 | +(`proxmox://<SMBIOS UUID>`; the CCM runs in its `capmox` mode for that), the |
| 283 | +`topology.kubernetes.io/region|zone` labels, and no |
| 284 | +`node.cloudprovider.kubernetes.io/uninitialized` taint. |
| 285 | + |
| 286 | +What is covered by the integration and was verified live on PVE 9.2: |
| 287 | +PVC create → attach → online resize → delete; Cluster Autoscaler scale-up and |
| 288 | +scale-down through the pool; rolling replacement of a worker; |
| 289 | +MachineHealthCheck. |
| 290 | + |
| 291 | +## Isolation: what the design gives and what it does not |
| 292 | + |
| 293 | +- The hypervisor token is composed into the CCM and CSI controllers in the |
| 294 | + management cluster; nothing copies it into the tenant cluster. |
| 295 | +- The tenant is cluster-admin of its own cluster, and the controllers act on |
| 296 | + what its objects say: the CSI controller attaches the disk a PV names to the |
| 297 | + VM a Node names. **Scope comes from the Proxmox ACL**, not from the |
| 298 | + controllers: one pool and one storage per tenant. |
| 299 | +- A storage shared between tenants cannot be scoped: `Datastore.Allocate` on |
| 300 | + it reaches every volume on it. |
| 301 | +- capmox itself (VM creation) currently uses one platform-wide token for all |
| 302 | + tenants; per-tenant `credentialsRef` on `ProxmoxCluster` is being worked on |
| 303 | + upstream. |
| 304 | + |
| 305 | +## Limitations (v1.7) |
| 306 | + |
| 307 | +- Opt-in, experimental; no release gate yet (an e2e lane exists upstream, but its nightly run is opt-in and currently skipped). |
| 308 | +- Workers only. Cozystack's `vm-disk` / `vm-instance` applications still run on KubeVirt; a Proxmox substrate for them is in design. |
| 309 | +- Tenant `Kubernetes` kinds are published as part of the KubeVirt variant, so a management cluster still needs the KubeVirt packages installed even if every pool runs on Proxmox. |
| 310 | +- No DHCP: capmox assigns static addresses from `ipv4Config` only. |
| 311 | +- Linked clones keep the template's base snapshot referenced; rotate templates by creating a new VMID with new tags and rolling the pool. |
| 312 | +- No live migration awareness: the CCM reports a VM it cannot see as gone and deletes the Node. |
| 313 | +- Cozystack's Terraform provider does not model `substrate`/`proxmox.*` yet; `terraform apply` on such a cluster overwrites them. |
| 314 | + |
| 315 | +## Troubleshooting |
| 316 | + |
| 317 | +| Symptom | Cause | Fix | |
| 318 | +|---|---|---| |
| 319 | +| `InfrastructureProvider` not installing, fetch error | `capmox-config` Secret missing or in the wrong namespace | create it in `cozy-cluster-api` | |
| 320 | +| every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` | |
| 321 | +| `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly | |
| 322 | +| Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS | |
| 323 | +| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n <tenant ns> logs deploy/kubernetes-<cluster>-pccm` | |
| 324 | +| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM | |
| 325 | +| VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` | |
| 326 | +| `TalosConfigTemplate.Spec is immutable` | apiserver address changed on an existing cluster | delete the template; the reconcile Job recreates it | |
| 327 | +| capmox stuck at `Waiting for Cluster Controller to set OwnerRef` | `cluster.x-k8s.io/managed-by` annotation on the `ProxmoxCluster` | not expected from the chart; report it | |
| 328 | + |
| 329 | +## See also |
| 330 | + |
| 331 | +- [Managed Kubernetes]({{% ref "/docs/v1.7/kubernetes" %}}) — the application this page extends |
| 332 | +- Cluster API provider for Proxmox: https://github.com/ionos-cloud/cluster-api-provider-proxmox |
| 333 | +- Proxmox CCM / CSI: https://github.com/sergelogvinov/proxmox-cloud-controller-manager, https://github.com/sergelogvinov/proxmox-csi-plugin |
| 334 | +- Support-tier proposal: https://github.com/cozystack/community/pull/98 |
0 commit comments