diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index f61fa6a..d183126 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "plugins": [ { "name": "cozystack", - "description": "Cozystack platform skills bundle — wizard (entry-point orchestrator that interviews + dispatches the chain), talos-bootstrap (Talos node prep via talm with maintenance-mode probe, cert-SAN NAT guardrail, multidoc machine-config, and opt-in boot-method picker), talos-reset (cloud-provider terminate+relaunch helper for unrecoverable Talos nodes — OCI/AWS/GCP/Hetzner; preserves disks + VNICs + NSGs), ubuntu-bootstrap (Ubuntu/Debian k3s bootstrap wrapping ansible-cozystack), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning, extractedprism HA proxy, all-HRs-Ready + storage-pools-registered gate), debug (investigate stuck installs — classify operator-error/config-drift/upstream-bug/not-supported, apply fixes or workarounds, draft upstream issues on approval), cluster-upgrade (release-notes-driven v1.x patch/minor upgrade), package-deploy (dev-loop deploy with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (Playwright + Vite dev server scaffolding for cozystack-ui pointed at a chosen kubeconfig — worktree, @playwright/test, Chromium, kubectl proxy + Vite on a free port), comment-audit (audit code comments in a diff for design-doc prose and report which to cut, which to keep, and which arguments are restated at several sites — reports only), comment-trim (apply the cuts and prove mechanically that only comments changed; leaves the result in the working tree). comment-audit and comment-trim work on any repository, not only Cozystack ones. Invoked as cozystack:wizard, cozystack:talos-bootstrap, cozystack:talos-reset, cozystack:ubuntu-bootstrap, cozystack:cluster-install, cozystack:debug, cozystack:cluster-upgrade, cozystack:package-deploy, cozystack:package-bump, cozystack:external-app-create, cozystack:dev-ui-bootstrap, cozystack:comment-audit, cozystack:comment-trim.", + "description": "Cozystack platform skills bundle — wizard (entry-point orchestrator that interviews + dispatches the chain), talos-bootstrap (Talos node prep via talm with maintenance-mode probe, cert-SAN NAT guardrail, multidoc machine-config, and opt-in boot-method picker), talos-reset (cloud-provider terminate+relaunch helper for unrecoverable Talos nodes — OCI/AWS/GCP/Hetzner; preserves disks + VNICs + NSGs), ubuntu-bootstrap (Ubuntu/Debian k3s bootstrap wrapping ansible-cozystack), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning, extractedprism HA proxy, all-HRs-Ready + storage-pools-registered gate), debug (investigate stuck installs — classify operator-error/config-drift/upstream-bug/not-supported, apply fixes or workarounds, draft upstream issues on approval), cluster-upgrade (release-notes-driven v1.x patch/minor upgrade), package-deploy (dev-loop deploy with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (Playwright + Vite dev server scaffolding for cozystack-ui pointed at a chosen kubeconfig — worktree, @playwright/test, Chromium, kubectl proxy + Vite on a free port), comment-audit (audit code comments in a diff for design-doc prose and report which to cut, which to keep, and which arguments are restated at several sites — reports only), comment-trim (apply the cuts and prove mechanically that only comments changed; leaves the result in the working tree). comment-audit and comment-trim work on any repository, not only Cozystack ones. Invoked as cozystack:wizard, cozystack:talos-bootstrap, cozystack:talos-reset, cozystack:ubuntu-bootstrap, cozystack:cluster-install, cozystack:debug, cozystack:cluster-upgrade, cozystack:package-deploy, cozystack:package-bump, cozystack:external-app-create, cozystack:dev-ui-bootstrap, cozystack:comment-audit, cozystack:comment-trim, cozystack:vm-tenant-move.", "source": "./plugins/cozystack", "category": "infrastructure" }, diff --git a/CLAUDE.md b/CLAUDE.md index 81e5bb9..080aa1b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,7 +28,7 @@ CLAUDE.md # this file — contributor guidance Two plugins ship today: -- `plugins/cozystack/` — platform bundle (13 skills: wizard, talos-bootstrap, talos-reset, ubuntu-bootstrap, cluster-install, debug, cluster-upgrade, package-deploy, package-bump, external-app-create, dev-ui-bootstrap, comment-audit, comment-trim). +- `plugins/cozystack/` — platform bundle (14 skills: wizard, talos-bootstrap, talos-reset, ubuntu-bootstrap, cluster-install, debug, cluster-upgrade, package-deploy, package-bump, external-app-create, dev-ui-bootstrap, comment-audit, comment-trim, vm-tenant-move). - `plugins/linstor/` — storage-recovery (1 skill: recover). Multi-skill plugin shape: every plugin has one `.claude-plugin/plugin.json` at its root, and one directory per skill under `skills/`. Skills are addressed by Claude Code as `/:` (e.g. `/cozystack:wizard`). diff --git a/README.md b/README.md index 4ff0701..53d96e5 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Install a plugin: ### cozystack -Platform skills bundle. One install gives you thirteen skills, invoked as `/cozystack:`. Start with `/cozystack:wizard` — it asks Talos / Ubuntu / Existing and picks the chain. +Platform skills bundle. One install gives you fourteen skills, invoked as `/cozystack:`. Start with `/cozystack:wizard` — it asks Talos / Ubuntu / Existing and picks the chain. | Skill | Description | | --- | --- | @@ -39,6 +39,7 @@ Platform skills bundle. One install gives you thirteen skills, invoked as `/cozy | **/cozystack:dev-ui-bootstrap** | Bootstrap a UI dev sandbox — Playwright + Vite dev server for `cozystack-ui` pointed at a chosen kubeconfig. Creates a feature worktree, installs `@playwright/test` and Chromium, writes `playwright.config.ts`, adds `dev:e2e` / `test:e2e` scripts, and brings up `kubectl proxy` + Vite on a free port. Use when fixing, debugging, or writing tests against the Cozystack console. | | **/cozystack:comment-audit** | Audit a change's code comments for design-doc prose — product rationale, before/after narrative, incident retelling, editorializing — and report which comments to cut, which earn their place, and which arguments are restated at several sites. Reports only; modifies nothing. Repository-agnostic: works on any codebase, not just Cozystack ones. | | **/cozystack:comment-trim** | Apply a comment audit — delete the prose, keep every comment that documents a real mechanism, replace sync-with-the-design-doc comments with a pointer, then prove with the bundled `comments-only.py` that no code moved. Leaves the edits in the working tree; does not commit or push. Repository-agnostic. | +| **/cozystack:vm-tenant-move** | Move a VM (VMInstance plus its VMDisks) to another tenant namespace without copying disk data. Prepares and pauses the VMDisks in the target, re-binds each PersistentVolume to a new PVC there, lets CDI adopt it through the `cdi.kubevirt.io/storage.populatedFor` annotation, and keeps the kube-ovn IP and MAC. Every mutation is confirmed per step; rollback is the same procedure in reverse. | Chains the wizard builds: @@ -73,7 +74,7 @@ Operators can opt out with `--no-extractedprism` and supply their own `--api-hos ```text plugins/ - cozystack/ # platform bundle (13 skills) + cozystack/ # platform bundle (14 skills) .claude-plugin/plugin.json skills/ wizard/ # entry point: interview + chain dispatcher @@ -89,6 +90,7 @@ plugins/ dev-ui-bootstrap/ # bootstrap Playwright + Vite sandbox for cozystack-ui comment-audit/ # report design-doc prose in code comments comment-trim/ # apply the cuts + prove comments-only + vm-tenant-move/ # move a VM to another tenant without copying disks linstor/ # storage bundle (1 skill) .claude-plugin/plugin.json skills/ diff --git a/plugins/cozystack/.claude-plugin/plugin.json b/plugins/cozystack/.claude-plugin/plugin.json index 0bac141..0b4a378 100644 --- a/plugins/cozystack/.claude-plugin/plugin.json +++ b/plugins/cozystack/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "cozystack", - "version": "1.18.1", - "description": "Cozystack platform skills bundle. Start with cozystack:wizard — it begins with a free-form 'tell me about your setup and goal' question, parses hints, then asks Talos / Ubuntu / Existing, builds a chain, and dispatches downstream skills via a cluster config directory the operator picks (every artifact lives there: inventory.yml, kubeconfig, .state.yaml, cozystack-platform-package.yaml — operator manages git on their own; optional sops opt-in encrypts secret files in-tree). Skills, invoked as cozystack:: wizard (orchestrator + 3-route dispatcher + Phase 4.5 active research + auto-dispatches debug on any failed_at), talos-bootstrap (Talos node prep via talm — Talos-1.12-aware maintenance probe, NAT-provider cert-SAN guardrail before first talm apply, multidoc machine-config with per-node VIP-link IPv4 stubs, etcd bootstrap, kubeconfig fetch, cozystack-tuned shape verification with Phase 11.5 auto-upgrade), talos-reset (cloud-provider terminate+relaunch helper for OCI/AWS/GCP/Hetzner when nodes are unrecoverable from inside; preserves block volumes + secondary VNICs + NSG memberships), ubuntu-bootstrap (wraps cozystack/ansible-cozystack — OS prep + k3s install in one go), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning via privileged DaemonSet on Talos with hostNetwork, extractedprism for kube-apiserver HA, OCI-tag-normalized cozy-installer chart, Platform Package, inline tenants/root ingress patch + LINSTOR pool registration during watch loop with combined HRs-Ready + pools-registered gate, Phase 8.6 default StorageClasses for v1.3.x, Phase 9.1 end-to-end reachability probe), debug (investigate a stuck or broken install — gathers symptoms, classifies operator error / config drift / upstream bug / not-yet-supported, applies fixes or workarounds, drafts upstream issues on approval; never opens PRs or files silently), cluster-upgrade (v1.x patch/minor upgrade with release-notes analysis), package-deploy (dev-loop deploy of a single package with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (bootstrap a UI dev sandbox — Playwright + Vite dev server for cozystack-ui pointed at a chosen kubeconfig; worktree, @playwright/test, Chromium, playwright.config.ts, dev:e2e / test:e2e scripts, kubectl proxy + Vite on a free port), comment-audit (audit a diff's code comments for design-doc prose — rationale, product motivation, before/after narrative, incident retelling — and report cuts, tightens, keeps and arguments duplicated across sites; reports only, modifies nothing), comment-trim (apply the cuts — delete the prose, tighten verbose-but-legitimate blocks, keep every comment that documents a real mechanism, then prove mechanically with the bundled comments-only.py that no code moved and a grep-based pointer-invariant check that no pointer sits on top of the prose it replaces; leaves the result in the working tree, does not commit or push). comment-audit and comment-trim are repository-agnostic and usable on any codebase. All skills match the operator's natural language detected from conversation context — code identifiers, commands, file paths, and GitHub-public text stay canonical. All skills follow the same gate-and-confirm discipline: read-only lookups run freely; any mutation needs explicit per-step approval.", + "version": "1.19.0", + "description": "Cozystack platform skills bundle. Start with cozystack:wizard — it begins with a free-form 'tell me about your setup and goal' question, parses hints, then asks Talos / Ubuntu / Existing, builds a chain, and dispatches downstream skills via a cluster config directory the operator picks (every artifact lives there: inventory.yml, kubeconfig, .state.yaml, cozystack-platform-package.yaml — operator manages git on their own; optional sops opt-in encrypts secret files in-tree). Skills, invoked as cozystack:: wizard (orchestrator + 3-route dispatcher + Phase 4.5 active research + auto-dispatches debug on any failed_at), talos-bootstrap (Talos node prep via talm — Talos-1.12-aware maintenance probe, NAT-provider cert-SAN guardrail before first talm apply, multidoc machine-config with per-node VIP-link IPv4 stubs, etcd bootstrap, kubeconfig fetch, cozystack-tuned shape verification with Phase 11.5 auto-upgrade), talos-reset (cloud-provider terminate+relaunch helper for OCI/AWS/GCP/Hetzner when nodes are unrecoverable from inside; preserves block volumes + secondary VNICs + NSG memberships), ubuntu-bootstrap (wraps cozystack/ansible-cozystack — OS prep + k3s install in one go), cluster-install (Cozystack on a ready cluster — node-readiness, ZFS pool provisioning via privileged DaemonSet on Talos with hostNetwork, extractedprism for kube-apiserver HA, OCI-tag-normalized cozy-installer chart, Platform Package, inline tenants/root ingress patch + LINSTOR pool registration during watch loop with combined HRs-Ready + pools-registered gate, Phase 8.6 default StorageClasses for v1.3.x, Phase 9.1 end-to-end reachability probe), debug (investigate a stuck or broken install — gathers symptoms, classifies operator error / config drift / upstream bug / not-yet-supported, applies fixes or workarounds, drafts upstream issues on approval; never opens PRs or files silently), cluster-upgrade (v1.x patch/minor upgrade with release-notes analysis), package-deploy (dev-loop deploy of a single package with ExternalArtifact support), package-bump (single-package version bump with changelog adaptation), external-app-create (scaffold a new external-apps package), dev-ui-bootstrap (bootstrap a UI dev sandbox — Playwright + Vite dev server for cozystack-ui pointed at a chosen kubeconfig; worktree, @playwright/test, Chromium, playwright.config.ts, dev:e2e / test:e2e scripts, kubectl proxy + Vite on a free port), comment-audit (audit a diff's code comments for design-doc prose — rationale, product motivation, before/after narrative, incident retelling — and report cuts, tightens, keeps and arguments duplicated across sites; reports only, modifies nothing), comment-trim (apply the cuts — delete the prose, tighten verbose-but-legitimate blocks, keep every comment that documents a real mechanism, then prove mechanically with the bundled comments-only.py that no code moved and a grep-based pointer-invariant check that no pointer sits on top of the prose it replaces; leaves the result in the working tree, does not commit or push), vm-tenant-move (move a VMInstance and its VMDisks to another tenant namespace without copying disk data — re-binds each PV to a new PVC, lets CDI adopt it through the storage.populatedFor annotation, keeps the kube-ovn IP and MAC). comment-audit and comment-trim are repository-agnostic and usable on any codebase. All skills match the operator's natural language detected from conversation context — code identifiers, commands, file paths, and GitHub-public text stay canonical. All skills follow the same gate-and-confirm discipline: read-only lookups run freely; any mutation needs explicit per-step approval.", "author": { "name": "Cozystack", "url": "https://github.com/cozystack" diff --git a/plugins/cozystack/skills/vm-tenant-move/SKILL.md b/plugins/cozystack/skills/vm-tenant-move/SKILL.md new file mode 100644 index 0000000..7eec5e3 --- /dev/null +++ b/plugins/cozystack/skills/vm-tenant-move/SKILL.md @@ -0,0 +1,474 @@ +--- +name: vm-tenant-move +description: Move a Cozystack virtual machine (VMInstance plus its VMDisks) from one tenant namespace to another without copying disk data. The PersistentVolume stays where it is and is re-bound to a new PVC in the target namespace; a VMDisk created in the target is paused, its blank DataVolume is replaced by one over the moved PVC, and CDI adopts the PVC through the cdi.kubevirt.io/storage.populatedFor annotation. Keeps the VM's kube-ovn IP and MAC by moving its ip.kubeovn.io record. Needs cluster-admin (PersistentVolumes are cluster-scoped). Read-only lookups run freely; every mutation is shown and confirmed per step. +argument-hint: "--context= --from= --to= --vm=" +--- + +# cozystack:vm-tenant-move + +Work in reasoning mode. Use the phrasing `cozystack:vm-tenant-move`. Announce phase transitions: `cozystack:vm-tenant-move Phase N — `. + +> **Note on language in this SKILL.md** — every operator-facing prompt below is written in English for clarity. At runtime the skill matches the operator's natural language detected from prior conversation messages. Code identifiers, commands, file paths, and any text destined for GitHub stay canonical regardless of operator language. + +## When to use this skill + +The operator wants an existing VM to live in another tenant: a customer reorganised its tenants, a VM was created in the wrong one, or a workload is handed to another team. The disks are large enough that copying them is not attractive, and a short downtime is acceptable. + +**Not for**: + +- Copying a VM while the original keeps running → restore a backup into another namespace instead (see *Fallback* at the end). +- Moving between clusters → backup and restore only. +- Renaming a VM inside the same namespace → out of scope. + +## How it works + +Disk data is never copied. Each disk's PersistentVolume keeps its data and only changes which PVC points at it: + +1. In the target namespace, create the `VMDisk` first, wait for its blank DataVolume, suspend its HelmRelease, and delete the blank DataVolume. +2. Stop the source VM, set its PVs to `Retain`, delete the source `VMInstance` and `VMDisk`. The PVCs go away; the PVs stay `Released` with the data. +3. In the target namespace, create a PVC `vm-disk-` with `spec.volumeName` and the annotation `cdi.kubevirt.io/storage.populatedFor: vm-disk-`, and replace the PV's old `claimRef` with a reference to it. The PV itself is not deleted. +4. Re-create the DataVolume over that PVC with the same Helm metadata the chart gave it, then resume the HelmRelease. +5. Move the VM's `ip.kubeovn.io` record, create the `VMInstance` halted, check its IP/MAC, start it. + +KubeVirt backend storage moves the same way. With `firmware.efiPersistent: true` KubeVirt keeps the EFI NVRAM in a separate PVC labelled `persistent-state-for=vm-instance-`, owned by the VirtualMachine and absent from `spec.disks` (`pkg/storage/backend-storage/backend-storage.go` in KubeVirt v1.9.0). Deleting the VMInstance garbage-collects that PVC, so its PV gets `Retain` together with the disks, and a PVC with the same label is bound to it in the target. KubeVirt looks the PVC up by that label and reuses it. + +Why `populatedFor` is enough: CDI v1.66.1 lets a PVC exist before its DataVolume when `pvc.Annotations["cdi.kubevirt.io/storage.populatedFor"] == dv.Name` (`pkg/controller/common/util.go`, `ClaimMayExistBeforeDataVolume`). The DataVolume webhook then accepts the DataVolume, the controller skips import/clone (`pvcRequiresWork`), marks the DataVolume `storage.prePopulated` and takes ownership of the PVC. No feature gate and no CDI configuration change is needed. `cdi.kubevirt.io/allowClaimAdoption` does **not** work on a PVC — CDI reads it from the DataVolume only — which is why older Cozystack procedures that put it on the PVC had to disable the CDI webhooks. + +The same PV re-bind is what Cozystack itself does in `internal/migrationcontroller/handoff.go` (VM import), and the IP/MAC move follows platform migration `packages/core/platform/images/migrations/migrations/29`. + +Names: `VMDisk ` → HelmRelease, DataVolume and PVC `vm-disk-`; `VMInstance ` → HelmRelease and VirtualMachine `vm-instance-`. + +## What changes for the VM owner + +Tell the owner before Phase 3: + +- Downtime lasts from Phase 3 until Phase 8. +- IP and MAC are kept (Phase 6) if both namespaces use the same kube-ovn subnet. The external LoadBalancer IP changes unless the cluster uses MetalLB and Phase 7's optional step is taken. +- **cloud-init runs as on a first boot.** The vm-instance chart derives the firmware UUID from the namespace (`stableUuid` in `packages/apps/vm-instance/templates/_helpers.tpl`), and KubeVirt passes it to the guest as the cloud-init instance-id. Expect new SSH host keys and user-data applied again. +- Service DNS name, network policies, quotas and backups are now the target tenant's. Old `Backup` objects stay in the source namespace; `Plan`s must be re-created. + +## Core principles + +- Match the operator's natural language. +- Every `kubectl` call carries an explicit `--context "$CTX"`. Never switch the kubeconfig's current context. +- Read-only lookups run freely. Every mutating command is printed and runs only after the operator confirms that step. +- Do not advance past a step until its **Done when** criterion holds. On any mismatch, stop and report — do not improvise a fix that deletes something. +- Never delete a PV. Never return a PV to `Delete` before Phase 9. +- Keep `$WORK` (saved manifests, `pv-map.txt` and `state-pv.txt`): it is the only input for a rollback. + +## Phase 1 — Scope, checks and snapshot + +```bash +export CTX='' +export SRC='tenant-' +export DST='tenant-' +export VM='' +export DISKS="$(kubectl --context "$CTX" -n "$SRC" get vminstance "$VM" -o jsonpath='{.spec.disks[*].name}')" +export WORK="$HOME/vm-move-$VM"; mkdir -p "$WORK" + +kubectl --context "$CTX" auth can-i patch persistentvolumes # yes +kubectl --context "$CTX" -n "$SRC" get vminstance "$VM" -o jsonpath='{.spec.networks}{"\n"}' +kubectl --context "$CTX" -n "$DST" get vminstance,vmdisk # $VM and $DISKS must not exist here +kubectl --context "$CTX" -n "$DST" describe resourcequota # room for the disks and the VM + +kubectl --context "$CTX" -n "$SRC" get vminstance "$VM" -o yaml > "$WORK/src-vminstance.yaml" +kubectl --context "$CTX" get ip.kubeovn.io "vm-instance-$VM.$SRC" -o jsonpath='{.spec.ipAddress} {.spec.macAddress} {.spec.subnet}' > "$WORK/src-ovn-ip.txt" +kubectl --context "$CTX" -n "$SRC" get svc "vm-instance-$VM" -o jsonpath='{.status.loadBalancer.ingress[0].ip}' > "$WORK/src-ext-ip.txt" +: > "$WORK/pv-map.txt" +for d in $DISKS; do + kubectl --context "$CTX" -n "$SRC" get vmdisk "$d" -o yaml > "$WORK/src-vmdisk-$d.yaml" + PV=$(kubectl --context "$CTX" -n "$SRC" get pvc "vm-disk-$d" -o jsonpath='{.spec.volumeName}') + REQ=$(kubectl --context "$CTX" -n "$SRC" get pvc "vm-disk-$d" -o jsonpath='{.spec.resources.requests.storage}') + POLICY=$(kubectl --context "$CTX" get pv "$PV" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}') + echo "$d $PV $REQ $POLICY" >> "$WORK/pv-map.txt" +done +cat "$WORK/pv-map.txt" + +# KubeVirt backend storage (persistent EFI NVRAM): not a VMDisk, owned by the VM +kubectl --context "$CTX" -n "$SRC" get vminstance "$VM" -o jsonpath='efiPersistent={.spec.firmware.efiPersistent}{"\n"}' +: > "$WORK/state-pv.txt" +for PVC in $(kubectl --context "$CTX" -n "$SRC" get pvc -l "persistent-state-for=vm-instance-$VM" -o jsonpath='{.items[*].metadata.name}'); do + kubectl --context "$CTX" -n "$SRC" get pvc "$PVC" -o yaml > "$WORK/src-state-pvc-$PVC.yaml" + PV=$(kubectl --context "$CTX" -n "$SRC" get pvc "$PVC" -o jsonpath='{.spec.volumeName}') + REQ=$(kubectl --context "$CTX" -n "$SRC" get pvc "$PVC" -o jsonpath='{.spec.resources.requests.storage}') + POLICY=$(kubectl --context "$CTX" get pv "$PV" -o jsonpath='{.spec.persistentVolumeReclaimPolicy}') + echo "$PVC $PV $REQ $POLICY" >> "$WORK/state-pv.txt" +done +cat "$WORK/state-pv.txt" +``` + +Also check that no other VMInstance in `$SRC` lists the same disks. + +**Done when** `pv-map.txt` has one ` ` line per disk and `$DST` has nothing with the same names. `state-pv.txt` has at most one ` ` line, and every line has a PV name: with `efiPersistent=true` and an empty file the VM never booted with persistent EFI and KubeVirt creates fresh NVRAM in the target; two lines, or a PVC without a PV, stop the procedure. If the VM has `networks`, the same NetworkAttachmentDefinitions must exist in `$DST`, or the operator decides to drop them from the VMInstance in Phase 7. + +If the cluster has Velero and a BackupClass, offer a `BackupJob` of the VMInstance before Phase 3 as insurance. + +## Phase 2 — Prepare the VMDisks in the target and pause them + +No downtime yet. For each disk create a `VMDisk` with the same `storage`, `storageClass` and `optical` as in `src-vmdisk-.yaml`, **without `source`** and with **`bindImmediately: true` even if the source has `false`**. With deferred binding on a WaitForFirstConsumer class the blank DataVolume has no consumer: CDI (`HonorWaitForFirstConsumer` is on in Cozystack) does not populate it without a selected node or the immediate-binding annotation (`pkg/controller/populators/util.go` in CDI v1.66.1), and the wait below times out. The moved PV is already provisioned, so the flag no longer affects it; Phase 9 restores the source value. + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: VMDisk +metadata: + name: + namespace: +spec: + storage: + storageClass: # must match the PV's StorageClass + optical: false + bindImmediately: true # always; see above +``` + +```bash +kubectl --context "$CTX" apply -f "$WORK/dst-vmdisk-$d.yaml" +kubectl --context "$CTX" -n "$DST" wait dv "vm-disk-$d" --for=jsonpath='{.status.phase}'=Succeeded --timeout=5m +kubectl --context "$CTX" -n "$DST" patch hr "vm-disk-$d" --type merge -p '{"spec":{"suspend":true}}' +kubectl --context "$CTX" -n "$DST" get dv "vm-disk-$d" -o yaml > "$WORK/dst-dv-$d.yaml" +kubectl --context "$CTX" -n "$DST" delete dv "vm-disk-$d" +kubectl --context "$CTX" -n "$DST" wait pvc "vm-disk-$d" --for=delete --timeout=5m +``` + +**Done when** every HelmRelease `vm-disk-` in `$DST` has `suspend: true`, its DataVolume YAML is saved, and the blank DataVolume and PVC are gone. The blank PV goes with them because its policy is still the class default. + +## Phase 3 — Stop the source VM and protect its volumes + +Downtime starts here. + +```bash +kubectl --context "$CTX" -n "$SRC" patch vminstance "$VM" --type merge -p '{"spec":{"runStrategy":"Halted"}}' +kubectl --context "$CTX" -n "$SRC" wait vmi "vm-instance-$VM" --for=delete --timeout=5m +cat "$WORK/pv-map.txt" "$WORK/state-pv.txt" | while read _ PV REQ POLICY; do + kubectl --context "$CTX" patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +done +``` + +**Done when** the VMI is gone and every PV in `pv-map.txt` and `state-pv.txt` shows `Retain`. **Do not continue without `Retain`**: Phase 4 would delete the data. + +## Phase 4 — Delete the VM and disks in the source tenant + +```bash +kubectl --context "$CTX" -n "$SRC" delete vminstance "$VM" +for d in $DISKS; do + kubectl --context "$CTX" -n "$SRC" delete vmdisk "$d" + kubectl --context "$CTX" -n "$SRC" wait pvc "vm-disk-$d" --for=delete --timeout=5m +done +while read PVC PV REQ POLICY; do + kubectl --context "$CTX" -n "$SRC" wait pvc "$PVC" --for=delete --timeout=5m # garbage-collected with the VM +done < "$WORK/state-pv.txt" +cat "$WORK/pv-map.txt" "$WORK/state-pv.txt" | while read _ PV REQ POLICY; do kubectl --context "$CTX" get pv "$PV"; done +``` + +**Done when** every PV from both files still exists, `Released`, `Retain`. Never delete a PVC by hand while its DataVolume exists: CDI may re-create it empty. + +## Phase 5 — Put the old PV under the target VMDisk + +The PVC copies `accessModes`, `volumeMode` and `storageClassName` from the PV. Pre-binding through `volumeName` skips the `volumeMode` match check, so a wrong mode binds and only fails at mount time. + +```bash +while read d PV REQ POLICY; do + SC=$(kubectl --context "$CTX" get pv "$PV" -o jsonpath='{.spec.storageClassName}') + MODE=$(kubectl --context "$CTX" get pv "$PV" -o jsonpath='{.spec.volumeMode}') + ACCESS=$(kubectl --context "$CTX" get pv "$PV" -o jsonpath='{.spec.accessModes}') + kubectl --context "$CTX" apply -f - <.yaml`. Keep `metadata.name`, `namespace`, `labels`, the `meta.helm.sh/*` and `vm-disk.cozystack.io/*` annotations and the whole `spec`. Drop `uid`, `resourceVersion`, `creationTimestamp`, `managedFields`, `status`, `ownerReferences` and every `cdi.kubevirt.io/*` annotation except `storage.bind.immediate.requested`: + +```yaml +apiVersion: cdi.kubevirt.io/v1beta1 +kind: DataVolume +metadata: + name: vm-disk- + namespace: + labels: + app.kubernetes.io/instance: vm-disk- + app.kubernetes.io/managed-by: Helm + annotations: + meta.helm.sh/release-name: vm-disk- + meta.helm.sh/release-namespace: + vm-disk.cozystack.io/source: "{}" + vm-disk.cozystack.io/storage-class: "" + vm-disk.cozystack.io/optical: "false" + cdi.kubevirt.io/storage.bind.immediate.requested: "true" # present, as Phase 2 set bindImmediately +spec: + # spec from dst-dv-.yaml, unchanged (blank source, storage) +``` + +```bash +kubectl --context "$CTX" apply -f "$WORK/new-dv-$d.yaml" +kubectl --context "$CTX" -n "$DST" get dv "vm-disk-$d" -o jsonpath='{.status.phase} {.metadata.annotations.cdi\.kubevirt\.io/storage\.prePopulated}{"\n"}' +kubectl --context "$CTX" -n "$DST" get pvc "vm-disk-$d" -o jsonpath='{.metadata.ownerReferences[0].kind}{"\n"}' +kubectl --context "$CTX" -n "$DST" get pods | grep -E 'importer|upload|clone' || echo "no CDI worker pods" +``` + +**Done when** the DataVolume is `Succeeded` with `prePopulated=vm-disk-`, the PVC is owned by a `DataVolume`, and no importer/upload/clone pod exists. + +- Webhook rejects with `Destination PVC … already exists` → the PVC lacks `populatedFor` or it names something else. Fix the annotation and apply again. +- An `importer-*` pod appears → stop, delete nothing, and report. + +Resume the HelmRelease: + +```bash +kubectl --context "$CTX" -n "$DST" patch hr "vm-disk-$d" --type merge -p '{"spec":{"suspend":false}}' +kubectl --context "$CTX" -n "$DST" wait hr "vm-disk-$d" --for=condition=Ready --timeout=5m +``` + +The chart finds the existing DataVolume and renders its `spec` as is (`packages/apps/vm-disk/templates/dv.yaml`, the `$existingDV` branch), so nothing is re-created. + +## Phase 6 — Move the IP and MAC (kube-ovn) + +kube-ovn keeps the VM's address in the cluster-scoped `ip.kubeovn.io` object `vm-instance-.`. When the vm-instance chart renders, it looks that object up for its own namespace and, if found, puts its IP and MAC into the `ovn.kubernetes.io/ip_address` and `ovn.kubernetes.io/mac_address` annotations of the VM template (`packages/apps/vm-instance/templates/vm.yaml`). Creating the record under the new name before the VMInstance exists is therefore enough. + +This only works if the target namespace is in the same subnet. By default every tenant uses `ovn-default`; if the namespace carries `ovn.kubernetes.io/logical_switch`, it must equal `$OVN_SUBNET`. If not, skip this phase and tell the owner the IP and MAC will change. + +```bash +read OVN_IP OVN_MAC OVN_SUBNET < "$WORK/src-ovn-ip.txt" +kubectl --context "$CTX" get ns "$DST" -o jsonpath='{.metadata.annotations.ovn\.kubernetes\.io/logical_switch}{"\n"}' + +kubectl --context "$CTX" apply -f - < "$WORK/dst-vminstance.yaml" +# set runStrategy: Halted in dst-vminstance.yaml and show it to the operator before applying +kubectl --context "$CTX" apply -f "$WORK/dst-vminstance.yaml" +kubectl --context "$CTX" -n "$DST" wait hr "vm-instance-$VM" --for=condition=Ready --timeout=5m +kubectl --context "$CTX" -n "$DST" get vm "vm-instance-$VM" -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' +``` + +If `state-pv.txt` is not empty, give the backend-storage PVC the owner the source one had, so it is garbage-collected with this VM later and a future VM of the same name does not inherit its NVRAM: + +```bash +VM_UID=$(kubectl --context "$CTX" -n "$DST" get vm "vm-instance-$VM" -o jsonpath='{.metadata.uid}') +while read PVC PV REQ POLICY; do + kubectl --context "$CTX" -n "$DST" patch pvc "$PVC" --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"kubevirt.io/v1\",\"kind\":\"VirtualMachine\",\"name\":\"vm-instance-$VM\",\"uid\":\"$VM_UID\",\"controller\":true,\"blockOwnerDeletion\":true}]}}" +done < "$WORK/state-pv.txt" +``` + +**Done when** the VM template carries `ovn.kubernetes.io/ip_address` and `ovn.kubernetes.io/mac_address` with the old values (if Phase 6 ran). If they are missing, the chart did not find the record from Phase 6: check its name. + +Optional, MetalLB only (`metallb.io/loadBalancerIPs`, present in the MetalLB v0.16.1 Cozystack ships) — keep the external IP: + +```bash +kubectl --context "$CTX" -n "$DST" annotate svc "vm-instance-$VM" metallb.io/loadBalancerIPs="$(cat "$WORK/src-ext-ip.txt")" +``` + +## Phase 8 — Start and verify + +```bash +kubectl --context "$CTX" -n "$DST" patch vminstance "$VM" --type merge -p '{"spec":{"runStrategy":"Always"}}' +kubectl --context "$CTX" -n "$DST" wait vmi "vm-instance-$VM" --for=condition=Ready --timeout=10m +virtctl --context "$CTX" -n "$DST" ssh @vm/vm-instance-$VM -c 'ip a; df -h; cloud-init status' +# with state-pv.txt: the VMI must use the moved PVC, not a new one +kubectl --context "$CTX" -n "$DST" get vmi "vm-instance-$VM" -o jsonpath='{.status.volumeStatus[?(@.name=="persistent-state-for-this-vm")].persistentVolumeClaimInfo.claimName}{"\n"}' +``` + +Use the original `runStrategy` from `src-vminstance.yaml` if it was not `Always`. + +**Done when** the guest booted, the data is there, the network is up, and the backend-storage claim (if any) is the PVC named in `state-pv.txt`. A new SSH host key is expected. + +No network on an Ubuntu guest without `cloudInit` and `sshKeys`: cloud-init did not re-run and netplan is still bound to the old MAC, so the MAC was not kept — re-check Phase 6, otherwise fix netplan from `virtctl console`. + +## Phase 9 — Close + +Only after the VM owner confirms the VM works. Return every PV to the policy recorded in Phase 1 (usually `Delete`): + +```bash +cat "$WORK/pv-map.txt" "$WORK/state-pv.txt" | while read _ PV REQ POLICY; do + kubectl --context "$CTX" patch pv "$PV" -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"$POLICY\"}}" +done +``` + +Restore `bindImmediately` on every disk whose source VMDisk had `false`. The chart then drops the immediate-binding annotation from the DataVolume; the PVC is already bound, so nothing is re-provisioned: + +```bash +for d in $DISKS; do + if grep -qE '^ bindImmediately: false$' "$WORK/src-vmdisk-$d.yaml"; then + kubectl --context "$CTX" -n "$DST" patch vmdisk "$d" --type merge -p '{"spec":{"bindImmediately":false}}' + kubectl --context "$CTX" -n "$DST" wait hr "vm-disk-$d" --for=condition=Ready --timeout=5m + fi +done +``` + +Then re-create backup `Plan`s in `$DST` and remove the old ones in `$SRC`. Keep `$WORK` as long as backups of the source VM exist. + +## Rollback + +While the PVs are `Retain`, no phase loses data. Only deleting a PV, or deleting a PVC while its PV is `Delete`, is irreversible. Pick the path by how far the move got; each one rebuilds from `$WORK`, never from the cluster state alone. + +A prepared VMDisk in `$DST` is removed by resuming its HelmRelease first: Flux skips `helm uninstall` for a suspended HelmRelease that is deleted, and the release would stay behind in `$DST`. + +```bash +remove_dst_vmdisk() { # $1 = disk name + kubectl --context "$CTX" -n "$DST" patch hr "vm-disk-$1" --type merge -p '{"spec":{"suspend":false}}' + kubectl --context "$CTX" -n "$DST" delete vmdisk "$1" + kubectl --context "$CTX" -n "$DST" wait pvc "vm-disk-$1" --for=delete --timeout=5m +} +``` + +### Before Phase 4 + +The source VMInstance and VMDisks still exist. Restore the source `runStrategy` from `src-vminstance.yaml`, return every PV in `pv-map.txt` and `state-pv.txt` to its recorded policy (the Phase 9 loop), and run `remove_dst_vmdisk` for each disk prepared in Phase 2. + +### After Phase 4, before Phase 7 is done + +The source VMInstance is gone and the target one does not exist yet (or Phase 7 failed half way), so the reverse procedure cannot start: its Phase 1 reads the disks from a VMInstance. Recover into `$SRC` from the saved manifests and the PV records instead. + +1. If Phase 7 created a VMInstance in `$DST`, delete it: `kubectl --context "$CTX" -n "$DST" delete vminstance "$VM"`. Its HelmRelease is not suspended, so the VM goes away and a backend-storage PVC it already owns is garbage-collected; the PV is `Retain`. + +2. Release every PV that is claimed in `$DST`. The HelmReleases are still suspended, so nothing re-creates what is deleted here. A DataVolume goes first: it owns its PVC, and a PVC deleted alone may be re-created empty by CDI. + + ```bash + for d in $DISKS; do + if kubectl --context "$CTX" -n "$DST" get dv "vm-disk-$d" >/dev/null 2>&1; then + kubectl --context "$CTX" -n "$DST" delete dv "vm-disk-$d" + else + kubectl --context "$CTX" -n "$DST" delete pvc "vm-disk-$d" --ignore-not-found + fi + kubectl --context "$CTX" -n "$DST" wait pvc "vm-disk-$d" --for=delete --timeout=5m + done + while read PVC PV REQ POLICY; do + kubectl --context "$CTX" -n "$DST" delete pvc "$PVC" --ignore-not-found --wait + done < "$WORK/state-pv.txt" + ``` + +3. Classify the disks. A disk whose PV is still `Bound` to `$SRC/vm-disk-` was not reached by Phase 4: its source VMDisk exists and needs nothing. Every other PV must be `Released` and goes into `rollback-map.txt`: + + ```bash + : > "$WORK/rollback-map.txt" + while read d PV REQ POLICY; do + STATE=$(kubectl --context "$CTX" get pv "$PV" -o jsonpath='{.status.phase} {.spec.claimRef.namespace}/{.spec.claimRef.name}') + echo "$d $PV: $STATE" + [ "$STATE" = "Bound $SRC/vm-disk-$d" ] || echo "$d $PV $REQ $POLICY" >> "$WORK/rollback-map.txt" + done < "$WORK/pv-map.txt" + cat "$WORK/rollback-map.txt" + ``` + + Stop if a PV is missing, or if one outside `rollback-map.txt` is in any state other than `Bound` to its source claim. + +4. Move the address back if Phase 6 ran: `kubectl --context "$CTX" delete ip.kubeovn.io "vm-instance-$VM.$DST" --ignore-not-found`. + +5. Re-run the move with the source namespace as the target, over the disks in `rollback-map.txt`: + + ```bash + export MOVED_TO="$DST" DST="$SRC" PVMAP="$WORK/rollback-map.txt" + ``` + + - Phase 2 for each disk in `$(cut -d' ' -f1 "$PVMAP")`. The restored VMDisk has no `source`, as it would have after the move. + - Phase 5: the disk loop reads `$PVMAP`; the backend-storage loop and the DataVolume re-creation run as written. + - Phase 6: only the `apply` of the record (it now creates `vm-instance-$VM.$SRC` from `src-ovn-ip.txt`); skip its `delete` line, which would remove the record just created. + - Phases 7–9 as written: Phase 7 rebuilds the VMInstance from `src-vminstance.yaml`, Phase 8 uses its original `runStrategy`, Phase 9 restores the policies from the full `pv-map.txt`. Before Phase 9 run `export DISKS="$(cut -d' ' -f1 "$PVMAP")"` so it touches only the re-created VMDisks. + +6. Remove the VMDisks left in the original target, now that no PV points there: `export DST="$MOVED_TO"`, then `remove_dst_vmdisk ` for each disk. Resuming a HelmRelease whose DataVolume was deleted renders a new blank disk, which goes away with the VMDisk. + +### After Phase 7 is done + +The target VMInstance exists halted and every target disk has its PVC `Bound` and DataVolume `Succeeded`. Run the procedure in reverse, from Phase 1, with `SRC` and `DST` swapped and a **new `$WORK`** (`$HOME/vm-move-$VM-back`): Phase 1 truncates `pv-map.txt`, and the original one is still the record of the source policies. If the forward Phase 9 has not run, the original `bindImmediately` values are in the first `$WORK`'s `src-vmdisk-.yaml`, not in the target VMDisks. Back in the source namespace the VM gets its original firmware UUID (the formula is deterministic), so cloud-init does not run a second time. + +## Fallback: restore a backup into the target namespace + +Use it when the PV cannot be re-bound, or when a copy is wanted while the source keeps running. Data goes through backup storage, which is slower; it needs Velero and a BackupClass with a VMInstance strategy. + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: RestoreJob +metadata: + name: move- + namespace: +spec: + backupRef: + name: + targetApplicationRef: + apiGroup: apps.cozystack.io + kind: VMInstance + name: + options: + targetNamespace: + failIfTargetExists: true + keepOriginalIpAndMac: false # defaults to true; keep false while the source VM is alive +``` + +Reference: `internal/backupcontroller/velerostrategy_controller.go` (`RestoreOptions`, `resolveRestoreTarget`) and `examples/backups/vmi/07-restore-to-copy.sh` in `cozystack/cozystack`.