commvault.com commvault.com
Solved

Hyper V backup failed only for one VM

T
Rank
Userlevel 2
Badge +8

i am having this issue wherein the VM level backup for this one VM is failing 

“The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. Please check the available free space and status of the volume being protected.” 

Other VMs are fine. we are on SP22 ; unfortunately this has just gone out of support; however we are booking at time to upgrade to SP24; although I am not sure if being on SP22 affects the backup as other VMs are working okay.  Although support did advise that if we do not upgrade; they would not be able to troubleshoot the issue further.  

icon

Best answer by Theseeker 29 April 2022, 09:39

View original
If you have a question or comment, please create a topic

13 replies

Damian Andre
Rank
Userlevel 7
Badge +23

Hey @Theseeker,

What version of Hyper-V is this?

Are the other VMs that are working attached to the same datastores? Is there an iSCSI controller on the VM, any IDE disks?

T
Rank
Userlevel 2
Badge +8

OS disk is in IDE and rest of the other disks are on SCSCI 

Guest Os is Windows 2008 R2

HyperV is windows 20212 R2 version 6.3.9600

 

“Are the other VMs that are working attached to the same datastores? Is there an iSCSI controller on the VM, any IDE disks?”

Yes there us another VM (Windows 2008R2) disk configuration is similar OS disk IDE and data disks are SCSCI and the same location  on Hyper-V server’s local disk.  All these disks then seem to have mount points which then point to a separate disk when I check the disk mgmt of the hyper-v server 

Damian Andre
Rank
Userlevel 7
Badge +23

OS disk is in IDE and rest of the other disks are on SCSCI 

Guest Os is Windows 2008 R2

HyperV is windows 20212 R2 version 6.3.9600

 

“Are the other VMs that are working attached to the same datastores? Is there an iSCSI controller on the VM, any IDE disks?”

Yes there us another VM (Windows 2008R2) disk configuration is similar OS disk IDE and data disks are SCSCI and the same location  on Hyper-V server’s local disk.  All these disks then seem to have mount points which then point to a separate disk when I check the disk mgmt of the hyper-v server 

My guess is that there issue with quiescing the guest OS if there is a near-identical VM backing up OK - the event logs in the guest OS should show more - alternative you could try a separate subclient with quiescing turned off (crash consistent backups) and see if that helps?

T
Rank
Userlevel 2
Badge +8

the customer has 3 hosts which are clustered using failover cluster (Failover cluster manager)

the disks are presented to these hosts as Clustered Shared Volume 

Looking from the disks view on the Failover cluster manager; these disks show minimal free space

 

 

 

The VM has which is eorring out after disabling quiescing has check-point location pointing to a CVFS volume which only has 80GB of free space 

the SQL server is is around 500 GB used (across all the disks) and has a total size of 1.2 TB apprxo disk attached to it.  Could not locate the minimal free space requirements 

T
Rank
Userlevel 2
Badge +8

I started getting this error after I disabled quiescing 

Jos Meijer
Rank
Userlevel 7
Badge +16

I would normally keep 10 to 15 percent free on the CSV.

If this is the issue you shouldn't be able to initiate a snapshot from hyper-v manager as well. Can you try to manually initiate a snapshot of the VM from within Hyper-v?

T
Rank
Userlevel 2
Badge +8

The snapshot process now works however the backup fails with error “The VSS shadow was lost while trying to read from disk [\\?\GLOBALROOT\Device\CSV{8e02e5ba-7691-4a7c-b2b0-e56676fb9309}\logs\server-sql_logs.vhd] for virtual machine [server-sql] on Host [SERVER-CL2]. This may indicate that there is insufficient COW space for the IO activity on the volume.”

the guest VM does have enough free space but as this is hyper-v cluster; however when I check the cluster manager; I notice disk at the hypervisor level is low on space 

 

T
Rank
Userlevel 2
Badge +8

also just wanted to get a clarity on the process of the VM level backup; I noticed that with HyperV v=backups a checkpoint gets created and then merged pretty quickly although backup is still progressing; also are the events logged on the active node with disks?  

christopherlecky
Rank
Userlevel 5
Badge +16

You will not be able to locate the minimum free space requirements because that won’t help you.

When the snapshot is initiated writes are deferred, until the snapshot is released.

This means that the amount of space the snapshot uses is dependant entirely on how many writes happen during that window. On super busy drives or drives that are moderately busy but extremely large the snapshot could be huge. 

I assume this is not connected to a SAN, because my first suggestion would be to simply expand the volume.

If that is not an option then you can actually move the place where the VSS snapshot holds data to another drive.

For very busy drives its best to defer the location to the fastest storage you have available. this will ensure that the drive can keep up with the writes and move the snapshot in a reasonable time.

 

Just google moving the vss snapshot.

I blame auto-correct for all misspellings, grammatical errors, and half-baked opinions.
christopherlecky
Rank
Userlevel 5
Badge +16

Looking closer it seems that you are doing non agent based SQL backups, you have to be careful there because if the SQL backups of SQL are happening during the snapshot window then you are working at cross purposes, the SQL backup would be growing the snapshot and the volume its writing to slowing things down coming and going.

 

you might want to look in SQL agent based backups, then you can simply reclaim the space dedicated to those volumes and kill two birds with one stone. 

I blame auto-correct for all misspellings, grammatical errors, and half-baked opinions.
T
Rank
Userlevel 2
Badge +8

The reason for doing a VM level backup is to migrate the customer’s VM to a VMWARE plat-form; the customer had a file system agent and SQL agent backup however the recovery of File agent and then recovering SQL databases on top is pretty cumbersome; so doing a VM level backup should make the migration pretty easy. 

 

As far as VSS snapshots it concerned; thanks for the suggestion and I googled it; in hyper-v if the CSV does not have enough free space; this error will occur (found the following on the Internet); there is no way to redirect snapshots on CSV.

“

Ok I worked on the snapshot issue I had and fixed some sharable points. We're talking about: clustered hyperv, CSV, backing up VMs...

1) the child partition snapshots is taken on the physical server, not onto VM
1a) All CSVs involved by VM get the "redirect access"
2) How much free space the physical server "see" on the csv is the point. If it is 100% full (for example due to a vdh) the snapshot cannot be saved (how the VM "see" it does not matter)

3) I did not find any way to address the snapshot volume elsewhere; normal tools (vssadmin, UI, ecc) do not allow to change volume

4) of course an hardware provider (instead of the software Microsoft one) could make the job easier, but more expensive

 

Summary:
Leave some free space for snapshots when allocating VM's VHDs

Ciao
Paolo”

 

T
Rank
Userlevel 2
Badge +8

the other thing was as the backup had finished; I had selected Crash Consistent on VM backup; if I choose File System and application consistent; would it interfere with the agent level SQL backups that are configured?  Like would it truncate the logs to databases with full recovery mode?

S
Rank
Userlevel 6
Badge +14

@Theseeker  No it should not cause any issues with the SQL agent. Those backups should still run their regular tlog backups and truncation

Terms & ConditionsCookie settings