Skip to content

Commit c6c4e9c

Browse files
committed
docs(kubernetes): make the Proxmox VE substrate enablement followable
The page used the capmox token in step 1 and created it in step 2, and showed bundles.enabledPackages as a values fragment without saying how to set it. Order the steps so each uses only what exists, give the Package patch, and add the CCM Secret the example references. Link the page from the deployment guide, which had no path to it. Carry the four v1.7 corrections from #782 to next/. v1.7.0 runs proxmox-cloud-controller-manager:v0.15.0, a tag that now resolves to a build that panics at start, so the v1.7 copy gets the image override until the backport in cozystack/cozystack#5150 is released. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
1 parent e17b6c6 commit c6c4e9c

4 files changed

Lines changed: 142 additions & 61 deletions

File tree

‎content/en/docs/next/install/_index.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,13 @@ If that's your case, we recommend using the guides below:
4545
- [Servers.com]({{% ref "/docs/next/install/providers/servers-com" %}})
4646

4747

48+
## Proxmox VE
49+
50+
There are two ways to use Cozystack with an existing Proxmox VE cluster, and they can be combined:
51+
52+
- Run the Cozystack cluster itself in Proxmox VE VMs: follow the [generic installation path](#generic-installation-path), treating the VMs like any other machines. Set the VM CPU type to `host` if tenants will run KubeVirt VMs on them.
53+
- Run the workers of tenant Kubernetes clusters as VMs on Proxmox VE instead of KubeVirt: [enable the Proxmox VE substrate]({{% ref "/docs/next/kubernetes/proxmox-substrate" %}}). This is opt-in and experimental.
54+
4855
## Upgrading and Post-deployment Configuration
4956

5057
After you've deployed a cluster, proceed to the [Cluster Administration]({{% ref "/docs/next/operations" %}}) section for

‎content/en/docs/next/kubernetes/proxmox-substrate.md‎

Lines changed: 58 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@ not yet covered by the project's release gate. Track the decision on support
1212
tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481).
1313
{{< /note >}}
1414

15+
This page is about running the **workers of tenant clusters** on Proxmox VE.
16+
To run the management cluster itself on Proxmox VE VMs, follow the regular
17+
[deployment guide]({{% ref "/docs/next/install" %}}).
18+
1519
## What it is
1620

1721
By default a managed (tenant) Kubernetes cluster in Cozystack has its control
@@ -49,6 +53,7 @@ The hypervisor token never enters the tenant cluster.
4953
| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 |
5054
| Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x |
5155
| capmox | v0.7.7 (shipped) | |
56+
| Proxmox CCM | v0.16.1 (shipped) | pinned by digest |
5257
| Talos | the version in `kubernetes-nodes` `talos.version` | template must match it |
5358
| Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested |
5459
| Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* |
@@ -70,22 +75,28 @@ On the **Proxmox VE** side:
7075

7176
On the **Cozystack** side:
7277

73-
- a working management cluster with the `iaas` bundle;
78+
- a working management cluster on a variant with the `iaas` bundle, such as
79+
`isp-full` or `isp-full-generic`;
7480
- a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address.
7581

76-
## 1. Enable the packages
82+
## 1. Enable the Proxmox provider
7783

78-
Add the Proxmox infrastructure provider to the platform's enabled packages.
79-
The in-cluster IPAM provider is pulled in as its dependency.
84+
The provider is an opt-in package, because it needs a Proxmox API token before
85+
it can do anything. On a Proxmox VE node, create a user and a token for capmox,
86+
following the provider's own
87+
[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md):
8088

81-
```yaml
82-
# ConfigMap cozystack in cozy-system, or the platform values
83-
bundles:
84-
enabledPackages:
85-
- cozystack.capi-provider-infra-proxmox
89+
```bash
90+
pveum user add capmox@pve
91+
pveum aclmod / -user capmox@pve -role PVEVMAdmin
92+
pveum user token add capmox@pve cozystack -privsep 0
8693
```
8794

88-
Before the provider is deployed, create the Secret it authenticates with.
95+
`PVEVMAdmin` on `/` is what the provider documents; a narrower set is
96+
described in its advanced-setup docs. This token is platform-wide: capmox
97+
clones the workers of every tenant with it (see *Isolation*).
98+
99+
Put the token into the Secret the provider authenticates with.
89100
capi-operator fetches it first; while it is missing nothing is installed and
90101
the `InfrastructureProvider` reports the fetch error.
91102

@@ -102,34 +113,41 @@ Three things that are easy to get wrong:
102113
- `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone.
103114
- a rotated token is picked up only after a restart of the capmox manager pod.
104115

105-
Check:
116+
Then add the provider to the platform's enabled packages. The in-cluster IPAM
117+
provider is pulled in as its dependency.
106118

107119
```bash
108-
kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders
109-
kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam'
120+
kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{
121+
"spec": {
122+
"components": {
123+
"platform": {
124+
"values": {
125+
"bundles": {
126+
"enabledPackages": ["cozystack.capi-provider-infra-proxmox"]
127+
}
128+
}
129+
}
130+
}
131+
}
132+
}'
110133
```
111134

112-
## 2. Prepare Proxmox: user, tokens, pool
135+
A merge patch replaces the whole list: if `bundles.enabledPackages` already has
136+
entries, repeat them here.
113137

114-
Create a user and tokens for the platform. Use **privilege-separated tokens**
115-
and grant the privileges to the user as well: Proxmox caps a token at its
116-
user's permissions.
117-
118-
For capmox itself, follow the provider's own
119-
[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md):
138+
Check:
120139

121140
```bash
122-
pveum user add capmox@pve
123-
pveum aclmod / -user capmox@pve -role PVEVMAdmin
124-
pveum user token add capmox@pve cozystack -privsep 0
141+
kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders
142+
kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam'
125143
```
126144

127-
`PVEVMAdmin` on `/` is what the provider documents; a narrower set is
128-
described in its advanced-setup docs. This token is platform-wide: capmox
129-
clones the workers of every tenant with it (see *Isolation*).
145+
## 2. Prepare a tenant: pool and tokens
130146

131147
For each **tenant** create a resource pool and a dedicated storage, and grant
132-
the tenant's CCM and CSI tokens their privileges on those paths only:
148+
the tenant's CCM and CSI tokens their privileges on those paths only. Use
149+
**privilege-separated tokens** and grant the privileges to the user as well:
150+
Proxmox caps a token at its user's permissions.
133151

134152
| Component | Privileges | Path |
135153
|---|---|---|
@@ -168,6 +186,7 @@ Workers are cloned from a Talos VM template that already exists on the
168186
hypervisor. Build it with the script shipped in the repository; it reads
169187
`talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so
170188
the template matches what the chart will render into the machine config.
189+
Run it from a checkout of the Cozystack version the management cluster runs.
171190

172191
```bash
173192
# prints the program it would run, for review
@@ -223,7 +242,13 @@ Nodes as `topology.kubernetes.io/region`. Note the URL here carries
223242
kubectl -n tenant-a create secret generic proxmox-csi \
224243
--from-literal=url='https://pve.example.com:8006/api2/json' \
225244
--from-literal=token_id='tenant-a@pve!csi' \
226-
--from-literal=token_secret='<secret>' \
245+
--from-literal=token_secret='<csi token secret>' \
246+
--from-literal=region='pve'
247+
248+
kubectl -n tenant-a create secret generic proxmox-ccm \
249+
--from-literal=url='https://pve.example.com:8006/api2/json' \
250+
--from-literal=token_id='tenant-a@pve!ccm' \
251+
--from-literal=token_secret='<ccm token secret>' \
227252
--from-literal=region='pve'
228253
```
229254

@@ -241,7 +266,8 @@ metadata:
241266
spec:
242267
cluster: px
243268
substrate: proxmox
244-
replicas: 2
269+
minReplicas: 2
270+
maxReplicas: 2
245271
resources:
246272
cpu: 4
247273
memory: 8Gi
@@ -269,7 +295,7 @@ are plainly present on the template.
269295
kubectl -n tenant-a get cluster,proxmoxcluster,machinedeployment,machine
270296
kubectl -n tenant-a get proxmoxmachines -o custom-columns='NAME:.metadata.name,VMID:.spec.virtualMachineID,NODE:.status.proxmoxNode,IP:.status.ipAddresses.net0.ipv4,STATE:.status.vmStatus'
271297
kubectl -n tenant-a get inclusterippools,ipaddresses.ipam.cluster.x-k8s.io
272-
kubectl -n tenant-a get deploy px-pccm px-pcsi-controller
298+
kubectl -n tenant-a get deploy kubernetes-px-pccm kubernetes-px-pcsi-controller
273299
274300
# tenant cluster
275301
kubectl --kubeconfig <tenant kubeconfig> get nodes -o wide
@@ -319,8 +345,8 @@ MachineHealthCheck.
319345
| every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` |
320346
| `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly |
321347
| Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS |
322-
| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n <tenant ns> logs deploy/<cluster>-pccm` |
323-
| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM; the CSI release depends on it |
348+
| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n <tenant ns> logs deploy/kubernetes-<cluster>-pccm` |
349+
| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM |
324350
| VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` |
325351
| `TalosConfigTemplate.Spec is immutable` | apiserver address changed on an existing cluster | delete the template; the reconcile Job recreates it |
326352
| capmox stuck at `Waiting for Cluster Controller to set OwnerRef` | `cluster.x-k8s.io/managed-by` annotation on the `ProxmoxCluster` | not expected from the chart; report it |

‎content/en/docs/v1.7/install/_index.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,13 @@ If that's your case, we recommend using the guides below:
4545
- [Servers.com]({{% ref "/docs/v1.7/install/providers/servers-com" %}})
4646

4747

48+
## Proxmox VE
49+
50+
There are two ways to use Cozystack with an existing Proxmox VE cluster, and they can be combined:
51+
52+
- Run the Cozystack cluster itself in Proxmox VE VMs: follow the [generic installation path](#generic-installation-path), treating the VMs like any other machines. Set the VM CPU type to `host` if tenants will run KubeVirt VMs on them.
53+
- Run the workers of tenant Kubernetes clusters as VMs on Proxmox VE instead of KubeVirt: [enable the Proxmox VE substrate]({{% ref "/docs/v1.7/kubernetes/proxmox-substrate" %}}). This is opt-in and experimental.
54+
4855
## Upgrading and Post-deployment Configuration
4956

5057
After you've deployed a cluster, proceed to the [Cluster Administration]({{% ref "/docs/v1.7/operations" %}}) section for

‎content/en/docs/v1.7/kubernetes/proxmox-substrate.md‎

Lines changed: 70 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@ not yet covered by the project's release gate. Track the decision on support
1212
tier in [cozystack/cozystack#3481](https://github.com/cozystack/cozystack/issues/3481).
1313
{{< /note >}}
1414

15+
This page is about running the **workers of tenant clusters** on Proxmox VE.
16+
To run the management cluster itself on Proxmox VE VMs, follow the regular
17+
[deployment guide]({{% ref "/docs/v1.7/install" %}}).
18+
1519
## What it is
1620

1721
By default a managed (tenant) Kubernetes cluster in Cozystack has its control
@@ -46,9 +50,10 @@ The hypervisor token never enters the tenant cluster.
4650

4751
| | Verified | Notes |
4852
|---|---|---|
49-
| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6 |
53+
| Cozystack | v1.7.0-rc.1 and `main` | the substrate is not in v1.6; v1.7.0 needs the CCM fix in step 4 |
5054
| Proxmox VE | 9.2.x | 8.x untested; the API calls used exist since 8.x |
5155
| capmox | v0.7.7 (shipped) | |
56+
| Proxmox CCM | v0.16.1 | v1.7.0 ships v0.15.0, which panics; see step 4 |
5257
| Talos | the version in `kubernetes-nodes` `talos.version` | template must match it |
5358
| Storage for workers | ZFS pool (linked clones verified) | any storage that supports clones should work; LVM-thin untested |
5459
| Storage for PVs | any Proxmox storage with `images` content | one storage per tenant recommended, see *Isolation* |
@@ -70,22 +75,28 @@ On the **Proxmox VE** side:
7075

7176
On the **Cozystack** side:
7277

73-
- a working management cluster with the `iaas` bundle;
78+
- a working management cluster on a variant with the `iaas` bundle, such as
79+
`isp-full` or `isp-full-generic`;
7480
- a MetalLB / LoadBalancer pool, since each tenant apiserver needs one address.
7581

76-
## 1. Enable the packages
82+
## 1. Enable the Proxmox provider
7783

78-
Add the Proxmox infrastructure provider to the platform's enabled packages.
79-
The in-cluster IPAM provider is pulled in as its dependency.
84+
The provider is an opt-in package, because it needs a Proxmox API token before
85+
it can do anything. On a Proxmox VE node, create a user and a token for capmox,
86+
following the provider's own
87+
[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md):
8088

81-
```yaml
82-
# spec.components.platform.values of the cozystack.cozystack-platform Package
83-
bundles:
84-
enabledPackages:
85-
- cozystack.capi-provider-infra-proxmox
89+
```bash
90+
pveum user add capmox@pve
91+
pveum aclmod / -user capmox@pve -role PVEVMAdmin
92+
pveum user token add capmox@pve cozystack -privsep 0
8693
```
8794

88-
Before the provider is deployed, create the Secret it authenticates with.
95+
`PVEVMAdmin` on `/` is what the provider documents; a narrower set is
96+
described in its advanced-setup docs. This token is platform-wide: capmox
97+
clones the workers of every tenant with it (see *Isolation*).
98+
99+
Put the token into the Secret the provider authenticates with.
89100
capi-operator fetches it first; while it is missing nothing is installed and
90101
the `InfrastructureProvider` reports the fetch error.
91102

@@ -102,34 +113,41 @@ Three things that are easy to get wrong:
102113
- `PROXMOX_TOKEN` is the full id `user@realm!tokenid`, not the token id alone.
103114
- a rotated token is picked up only after a restart of the capmox manager pod.
104115

105-
Check:
116+
Then add the provider to the platform's enabled packages. The in-cluster IPAM
117+
provider is pulled in as its dependency.
106118

107119
```bash
108-
kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders
109-
kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam'
120+
kubectl patch packages.cozystack.io cozystack.cozystack-platform --type=merge -p '{
121+
"spec": {
122+
"components": {
123+
"platform": {
124+
"values": {
125+
"bundles": {
126+
"enabledPackages": ["cozystack.capi-provider-infra-proxmox"]
127+
}
128+
}
129+
}
130+
}
131+
}
132+
}'
110133
```
111134

112-
## 2. Prepare Proxmox: user, tokens, pool
135+
A merge patch replaces the whole list: if `bundles.enabledPackages` already has
136+
entries, repeat them here.
113137

114-
Create a user and tokens for the platform. Use **privilege-separated tokens**
115-
and grant the privileges to the user as well: Proxmox caps a token at its
116-
user's permissions.
117-
118-
For capmox itself, follow the provider's own
119-
[usage guide](https://github.com/ionos-cloud/cluster-api-provider-proxmox/blob/main/docs/Usage.md):
138+
Check:
120139

121140
```bash
122-
pveum user add capmox@pve
123-
pveum aclmod / -user capmox@pve -role PVEVMAdmin
124-
pveum user token add capmox@pve cozystack -privsep 0
141+
kubectl -n cozy-cluster-api get infrastructureproviders,ipamproviders
142+
kubectl -n cozy-cluster-api get deploy | grep -E 'capmox|ipam'
125143
```
126144

127-
`PVEVMAdmin` on `/` is what the provider documents; a narrower set is
128-
described in its advanced-setup docs. This token is platform-wide: capmox
129-
clones the workers of every tenant with it (see *Isolation*).
145+
## 2. Prepare a tenant: pool and tokens
130146

131147
For each **tenant** create a resource pool and a dedicated storage, and grant
132-
the tenant's CCM and CSI tokens their privileges on those paths only:
148+
the tenant's CCM and CSI tokens their privileges on those paths only. Use
149+
**privilege-separated tokens** and grant the privileges to the user as well:
150+
Proxmox caps a token at its user's permissions.
133151

134152
| Component | Privileges | Path |
135153
|---|---|---|
@@ -168,6 +186,7 @@ Workers are cloned from a Talos VM template that already exists on the
168186
hypervisor. Build it with the script shipped in the repository; it reads
169187
`talos.version` and `talos.schematicID` from the `kubernetes-nodes` chart so
170188
the template matches what the chart will render into the machine config.
189+
Run it from a checkout of the Cozystack version the management cluster runs.
171190

172191
```bash
173192
# prints the program it would run, for review
@@ -223,13 +242,34 @@ Nodes as `topology.kubernetes.io/region`. Note the URL here carries
223242
kubectl -n tenant-a create secret generic proxmox-csi \
224243
--from-literal=url='https://pve.example.com:8006/api2/json' \
225244
--from-literal=token_id='tenant-a@pve!csi' \
226-
--from-literal=token_secret='<secret>' \
245+
--from-literal=token_secret='<csi token secret>' \
246+
--from-literal=region='pve'
247+
248+
kubectl -n tenant-a create secret generic proxmox-ccm \
249+
--from-literal=url='https://pve.example.com:8006/api2/json' \
250+
--from-literal=token_id='tenant-a@pve!ccm' \
251+
--from-literal=token_secret='<ccm token secret>' \
227252
--from-literal=region='pve'
228253
```
229254

230255
`substrate` is immutable: switching an existing cluster is refused. Create a
231256
new one.
232257

258+
{{< warning >}}
259+
Cozystack v1.7.0 ships a broken CCM image. The chart runs
260+
`proxmox-cloud-controller-manager:v0.15.0`, and that tag now resolves to a build
261+
that panics at start, so workers keep the `uninitialized` taint. The fix is in
262+
[cozystack/cozystack#5150](https://github.com/cozystack/cozystack/pull/5150) and
263+
is not released yet. Until then, set the image by hand once the cluster exists.
264+
The next Helm upgrade of the cluster puts the chart's image back, so repeat this
265+
after every change to the `Kubernetes` resource.
266+
{{< /warning >}}
267+
268+
```bash
269+
kubectl -n tenant-a set image deploy/kubernetes-px-pccm \
270+
cloud-controller-manager=ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.16.1@sha256:474dbfd3be5c2e0866e520bc6ddb60b7accef554ca7401488eee35252cdf8734
271+
```
272+
233273
## 5. Add a worker pool
234274

235275
```yaml
@@ -320,6 +360,7 @@ MachineHealthCheck.
320360
| every API call returns 501 | `PROXMOX_URL` already contains `/api2/json` | use bare `https://host:8006` |
321361
| `found 0 VM templates with tags ...` | `templateTags` ≠ the template's tag set | list every tag, exactly |
322362
| Machine `VMProvisioned=True`, node never joins, no CSR | workers cannot reach the tenant apiserver LoadBalancer IP, or no reachable `dnsServers` | fix L3 / DNS |
363+
| CCM in `CrashLoopBackOff`, `panic: version string "0.15.0"` | the image v1.7.0 ships | set the image as in step 4 |
323364
| Nodes `Ready` but keep `uninitialized` taint, pool `ScalingUp` forever | CCM not running or crash-looping | `kubectl -n <tenant ns> logs deploy/kubernetes-<cluster>-pccm` |
324365
| CSI node plugin `CrashLoopBackOff`, `Failed to get region or zone` | CCM has not labelled the node yet | wait for the CCM |
325366
| VM reboots every ~70 min, never joins | template built from the bare-metal `nocloud` image | rebuild with `hack/proxmox-talos-template.sh` |

0 commit comments

Comments
 (0)