What does “Unable/failed to quiesce the guest file system during snapshot creation” mean, and how do I resolve it on a VMware backup?

Hey @Vsicherman 

Getting Application Consistent snapshots is very important for the recoverability of files and application data on a VM. During a backup, Commvault instructs VMware to take a VSS-enabled snapshot, which will in-turn tell the file system and any applications to write any in-memory change to disk with a temporary I/O pause. Once the VSS snapshot is in place, VMware takes the virtual machine snapshot which ensures that all the data is captured in the most consistent form. 

Virtual Server backups may "Complete with Errors" when one or more virtual machine guest report quiescing error during a backup. Even though the VSS quiesced snapshot is failing, the data is still being protected as a Crash Consistent backup instead. A restore from this VM has the slight possibility of being inconsistent for in-use files. An an example, recovering this system would not be dissimilar if you were starting the VM after an ungraceful power outage OR a hardware failure. The data associated with this VM, if restored, for exmaple, a SQL database using its files, could result in a dirty SQL MDB that SQL would have to repair, possibility losing some transactions that were in-flight during the snapshot. 

You can view the VMs that failed to quiesce from the Backup History for the subclient -> right click on the Job that Completed with Errors -> View Job details -> Virtual Machine Status. Sort the list of VMs by Status, you should see Partial Success and Failure reason "Unable to quiesce". 

When Commvault creates a snapshot, it is done through the VMware API, as such, testing the ability to quiesce the virtual machine can be done by taking a snapshot through VMware and checking the quiesce box (although don’t snapshot the VM memory). If you can take a quiesced snapshot through VMware, Commvault should be able to do the same. If a quiesced snapshot can't be taken on the machine(s) displaying the warning, you should see this reflected in the VMware console which is why Commvault displays the warning. If you are unable to take a quiesced snapshot, here are some things you can try: 

1. Verify that VMware Tools are up to date on the problematic VMs. If they are not up to date, update them, and run a new backup and see if this resolves the issue. If it still fails, move on to step 4. 

2. Review the Windows Application System and Event logs on the Guest for Volume Shadow Copy service errors, or something similar. 

• Additionally, you can identify the specific failed VSS writer by issuing the command vssadmin list writers from an elevated command prompt. This command will list all of the VSS Writers on the system and their current status, ALL listed writers should show a Stable and No error listed under State and Last Error. If you see errors or a timeout status, a reboot should resolve the issue, although it may only be temporary if there is another underlying cause. 

Additionally, the load or I/O on the ESX Host and Datastore play a crucial role in the successful snapshot process, multiple concurrent snapshots can time out and generate these failures. You can try to consolidate the number of running jobs and control parallel snapshots using readers. 

If you are unable to resolve the issue with these steps, here are some other measures to help troubleshoot: 

• Can a quiesced snapshot be generated in VMware outside of Commvault? 

• Are there errors being generated in the 'events and tasks' of the VMware console during the backups? 

• Are the machines up to date with VMware tools? Are there VSS errors occurring in the machines' event viewer logs? 

• Is this a particularly large VM? 

• Can you test this machine in a separate sub client to see if the issue occurs when it runs by itself? 

• Do these machines have databases (SQL, Exchange, etc) that need to be quiesced, as a standard file system can simply use crash consistent?

-Cheers