[BUG] Scaling down a VMware-backed machine can leave an orphan powered-on VM when the configured ESX host was removed

[davidhrbac]

Rancher Server Setup

• Rancher version: v2.14.1
• Installation option (Docker install/Helm Chart):
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):
• Proxy/Cert Details:

Information about the Cluster

• Kubernetes version:
• Cluster Type (Local/Downstream): Downstream
  • If downstream, what type of cluster? Infrastructure Provider = VMWare builtin provider

User Information

• What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
  • Admin

Describe the bug
When scaling down a machine from a VMware-backed machine pool, Rancher can remove the node from the cluster while leaving the backing VM in VMware powered on and not deleted.

After that:

• the node is no longer part of the cluster
• the VM is still running in VMware
• Rancher no longer tracks that VM as part of the cluster

This was observed after a VMware infrastructure refresh where new ESX hosts were introduced and some older ESX hosts were removed. Existing Rancher cluster machine pool configurations still referenced one of the removed ESX hosts in the VMware hostsystem setting.

This has been observed repeatedly across multiple Rancher instances and does not appear to be isolated to a single environment. The common factor has been a broken or stale VMware machine pool configuration.

This was observed in the Cluster Management machine pools view of a provisioning cluster, for example:
https://<rancher-host>/dashboard/c/_/manager/provisioning.cattle.io.cluster/<namespace>/<cluster-id>#machine-pools

To Reproduce

1. Create a downstream RKE2 cluster that uses the built-in VMware infrastructure provider.
2. Configure a machine pool with a specific VMware hostsystem value instead of any.
3. Later, replace part of the VMware infrastructure so that the configured ESX host is removed and new ESX hosts are added.
4. Leave the Rancher machine pool configuration unchanged so it still references the removed ESX host.
5. In Cluster Management, open the cluster detail view and go to the Machine Pools tab:
   https://<rancher-host>/dashboard/c/_/manager/provisioning.cattle.io.cluster/<namespace>/<cluster-id>#machine-pools
6. Scale down one of the machines in that pool.
7. Observe the node disappearing from the cluster, but the corresponding VMware VM remaining powered on and not deleted.

Result

• The Kubernetes/Rancher node is removed from the cluster view.
• The VMware VM remains powered on.
• The VM is not deleted.
• Rancher no longer tracks that VM as an active cluster machine.

Expected Result
Scaling down a machine should also delete the corresponding VMware VM, even if the current machine pool configuration has become stale.

Machine deletion should not depend on current placement settings such as hostsystem, resource pool, network, or datastore values that may have changed or become invalid after the machine was originally created.

Screenshots

Additional context
This looks related to the VMware machine delete path re-validating current machine pool configuration during deletion.

Relevant code paths:

• shell/dialog/ScaleMachineDownDialog.vue
  • scale-down of an RKE2/CAPI machine marks the machine for deletion and reduces the pool quantity
• pkg/controllers/capr/machineprovision/controller.go
  • Rancher machine deletion uses the machineprovision controller
• pkg/controllers/capr/machineprovision/args.go
  • delete path invokes Rancher Machine rm with updated config
• machine/drivers/vmwarevsphere/vsphere.go
  • the VMware driver delete path calls Remove()
• machine/drivers/vmwarevsphere/create.go
  • preCreate() re-resolves current datacenter/network/hostsystem/resource pool values

Because of that, deletion appears to depend on the current VMware machine pool configuration instead of deleting the already-created VM by its persisted machine identity.

This issue may be related to previous VMware orphan VM cleanup issues, but this scenario specifically involves scale-down/delete of an existing machine when the current pool configuration is stale, not only create-failure cleanup:

• [BUG] vsphere 8 using rke2/k3s - if vsphere DRS maxVcpusPerCore limit is reached on host, rancher does not clean up vsphere VMs #44212
• [BUG] Misconfigured Node provisioned on VMWare fails to delete failed node #46210

[davidhrbac]

I prepared a proposed fix in rancher/machine for review:

• Relax vSphere delete prechecks machine#441

Summary of the approach:

• add a delete-specific vSphere initialization path (preRemove())
• keep the datacenter/finder setup needed for VM lookup and ISO cleanup
• stop validating current network, hostsystem, and resource pool settings during delete

Rationale:

• delete of an already-created VM should not depend on current placement config values that may have become stale after the machine was created
• this directly targets the reported scenario where hostsystem still points to an ESX host that has since been removed