Unable to delete Instance snapshot in "Error' state on stopped Instances

[rajujith]

problem

When instance snapshots fail to create on KVM due to various reasons like ' Guest agent is not responding' or 'internal snapshots of a VM with pflash based firmware require QCOW2 nvram format' the instance snapshot is created in CloudStack in 'Error' state. This can be deleted while the instance is 'Running' but when the instance is in 'Stopped' state, the instance snapshot deletion fails since the snapshot was never created in KVM.

On Management server:

2025-09-18 08:04:33,742 DEBUG [c.c.v.s.VMSnapshotManagerImpl] (Work-Job-Executor-14:[ctx-d7ea3fce, job-59/job-60, ctx-c90903e0]) (logid:f78a6e3a) Failed to create vm snapshot: [VMSnapshot {"id":2,"name":"i-2-6-VM_VS_20250918080427","uuid":"f5cadba9-b1ff-4436-8d34-0fb6768fcd29","vmId":6}] due to: Creating VM snapshot: i-2-6-VM_VS_20250918080427 failed due to  Create VM snapshot failed due to org.libvirt.LibvirtException: Operation not supported: internal snapshots of a VM with pflash based firmware require QCOW2 nvram format com.cloud.utils.exception.CloudRuntimeException: Creating VM snapshot: i-2-6-VM_VS_20250918080427 failed due to  Create VM snapshot failed due to org.libvirt.LibvirtException: Operation not supported: internal snapshots of a VM with pflash based firmware require QCOW2 nvram format
        at org.apache.cloudstack.storage.vmsnapshot.DefaultVMSnapshotStrategy.takeVMSnapshot(DefaultVMSnapshotStrategy.java:179)

KVM agent:

2025-09-18 08:05:49,263 DEBUG [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Executing command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ].
2025-09-18 08:05:49,274 WARN  [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Execution of process [76592] for command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ] failed.
2025-09-18 08:05:49,274 DEBUG [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Exit value of process [76592] for command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ] is [1].
2025-09-18 08:05:49,274 WARN  [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Process [76592] for command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ] encountered the error: [1].

Manual deletion attempt :

/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c
qemu-img: Could not delete snapshot 'i-2-6-VM_VS_20250918080427': snapshot not found

mysql> select * from vm_snapshots where id=2\G
*************************** 1. row ***************************
                 id: 2
               uuid: f5cadba9-b1ff-4436-8d34-0fb6768fcd29
               name: i-2-6-VM_VS_20250918080427
       display_name: sn2
        description: sn2
              vm_id: 6
         account_id: 2
          domain_id: 1
service_offering_id: 13
   vm_snapshot_type: DiskAndMemory
              state: Error
             parent: NULL
            current: NULL
       update_count: 4
            updated: 2025-09-18 08:05:49
            created: 2025-09-18 08:04:28
            removed: NULL

versions

4.20.1
KVM

The steps to reproduce the bug

1. Deploy a Windows instance where guest agent is not installed or configure vtpm.
2. Create an instance snapshot with memory, it should fail.
3. Stop the instance
4. Delete the instance snapshot in 'Error' state.

What to do about it?

Avoid creating an instance snapshot in 'Error' state or allow deletion.

[DaanHoogland]

This should be the fix. 958535ed3eaf.zip

@qumarissword69 , could you create this as a pull request, please?

[DaanHoogland]

This should be the fix. 958535ed3eaf.zip

@qumarissword69 , could you create this as a pull request, please?

no you can’t , you’re as dumb as what…

[weizhouapache]

@rajujith
can you share some logs before the logs you shared ?

I notice that ACS checks if vm snapshot exist before removing it

cloudstack/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtDeleteVMSnapshotCommandWrapper.java

Lines 100 to 110 in 8eb6ddd

	String qemuImgPath = Script.getExecutableAbsolutePath("qemu-img");
	List<String[]> commands = new ArrayList<>();
	commands.add(new String[]{qemuImgPath, "snapshot", "-l", sanitizeBashCommandArgument(rootDisk.getPath())});
	commands.add(new String[]{Script.getExecutableAbsolutePath("tail"), "-n", "+3"});
	commands.add(new String[]{Script.getExecutableAbsolutePath("awk"), "-F", " ", "{print $2}"});
	commands.add(new String[]{Script.getExecutableAbsolutePath("grep"), "^" + sanitizeBashCommandArgument(cmd.getTarget().getSnapshotName()) + "$"});
	String qemu_img_snapshot = Script.executePipedCommands(commands, 0).second();
	if (qemu_img_snapshot == null) {
	logger.info("Cannot find snapshot " + cmd.getTarget().getSnapshotName() + " in file " + rootDisk.getPath() + ", return true");
	return new DeleteVMSnapshotAnswer(cmd, cmd.getVolumeTOs());
	}

[rajujith]

@weizhouapache After upgrading the environment to 4.22-SNAPSHOT for another PR testing, the issue is not observed anymore.

[weizhouapache]

fixed by #11687