Hey @Theseeker,

What version of Hyper-V is this?

Are the other VMs that are working attached to the same datastores? Is there an iSCSI controller on the VM, any IDE disks?

OS disk is in IDE and rest of the other disks are on SCSCI 

Guest Os is Windows 2008 R2

HyperV is windows 20212 R2 version 6.3.9600

“Are the other VMs that are working attached to the same datastores? Is there an iSCSI controller on the VM, any IDE disks?”

Yes there us another VM (Windows 2008R2) disk configuration is similar OS disk IDE and data disks are SCSCI and the same location  on Hyper-V server’s local disk.  All these disks then seem to have mount points which then point to a separate disk when I check the disk mgmt of the hyper-v server 

My guess is that there issue with quiescing the guest OS if there is a near-identical VM backing up OK - the event logs in the guest OS should show more - alternative you could try a separate subclient with quiescing turned off (crash consistent backups) and see if that helps?

the customer has 3 hosts which are clustered using failover cluster (Failover cluster manager)

the disks are presented to these hosts as Clustered Shared Volume 

Looking from the disks view on the Failover cluster manager; these disks show minimal free space

The VM has which is eorring out after disabling quiescing has check-point location pointing to a CVFS volume which only has 80GB of free space 

the SQL server is is around 500 GB used (across all the disks) and has a total size of 1.2 TB apprxo disk attached to it.  Could not locate the minimal free space requirements 

I started getting this error after I disabled quiescing 

I would normally keep 10 to 15 percent free on the CSV.

If this is the issue you shouldn't be able to initiate a snapshot from hyper-v manager as well. Can you try to manually initiate a snapshot of the VM from within Hyper-v?

The snapshot process now works however the backup fails with error “The VSS shadow was lost while trying to read from disk k\\?\GLOBALROOT\Device\CSV{8e02e5ba-7691-4a7c-b2b0-e56676fb9309}\logs\server-sql_logs.vhd] for virtual machine eserver-sql] on Host tSERVER-CL2]. This may indicate that there is insufficient COW space for the IO activity on the volume.”

the guest VM does have enough free space but as this is hyper-v cluster; however when I check the cluster manager; I notice disk at the hypervisor level is low on space 

also just wanted to get a clarity on the process of the VM level backup; I noticed that with HyperV v=backups a checkpoint gets created and then merged pretty quickly although backup is still progressing; also are the events logged on the active node with disks?  

You will not be able to locate the minimum free space requirements because that won’t help you.

When the snapshot is initiated writes are deferred, until the snapshot is released.

This means that the amount of space the snapshot uses is dependant entirely on how many writes happen during that window. On super busy drives or drives that are moderately busy but extremely large the snapshot could be huge. 

I assume this is not connected to a SAN, because my first suggestion would be to simply expand the volume.

If that is not an option then you can actually move the place where the VSS snapshot holds data to another drive.

For very busy drives its best to defer the location to the fastest storage you have available. this will ensure that the drive can keep up with the writes and move the snapshot in a reasonable time.

Just google moving the vss snapshot.

Looking closer it seems that you are doing non agent based SQL backups, you have to be careful there because if the SQL backups of SQL are happening during the snapshot window then you are working at cross purposes, the SQL backup would be growing the snapshot and the volume its writing to slowing things down coming and going.

you might want to look in SQL agent based backups, then you can simply reclaim the space dedicated to those volumes and kill two birds with one stone. 

The reason for doing a VM level backup is to migrate the customer’s VM to a VMWARE plat-form; the customer had a file system agent and SQL agent backup however the recovery of File agent and then recovering SQL databases on top is pretty cumbersome; so doing a VM level backup should make the migration pretty easy. 

As far as VSS snapshots it concerned; thanks for the suggestion and I googled it; in hyper-v if the CSV does not have enough free space; this error will occur (found the following on the Internet); there is no way to redirect snapshots on CSV.

“

Ok I worked on the snapshot issue I had and fixed some sharable points. We're talking about: clustered hyperv, CSV, backing up VMs...

1) the child partition snapshots is taken on the physical server, not onto VM
1a) All CSVs involved by VM get the "redirect access"
2) How much free space the physical server "see" on the csv is the point. If it is 100% full (for example due to a vdh) the snapshot cannot be saved (how the VM "see" it does not matter)

3) I did not find any way to address the snapshot volume elsewhere; normal tools (vssadmin, UI, ecc) do not allow to change volume

4) of course an hardware provider (instead of the software Microsoft one) could make the job easier, but more expensive

Summary:
Leave some free space for snapshots when allocating VM's VHDs

Ciao
Paolo”

the other thing was as the backup had finished; I had selected Crash Consistent on VM backup; if I choose File System and application consistent; would it interfere with the agent level SQL backups that are configured?  Like would it truncate the logs to databases with full recovery mode?

@Theseeker  No it should not cause any issues with the SQL agent. Those backups should still run their regular tlog backups and truncation