Unable to quiesce guest file system during snapshot creation

https://kb.commvault.com/article/63085

A couple of ideas in this KB, but the main one is what Stuart mentioned (VMware Tools).

Thanks,
Scott
 

VSBKpWorker::UnmountVM() - Errors found during backup. Error string [unable to quiesce guest file system during snapshot creation]

This error come while Nutanix VM full backup running.

This warning is an issue between VMware and the Content(Guest OS) of the individual VMs and not a Commvault issue. As such Commvault is NOT the correct support team to troubleshoot further.
 
Just to explain the quiesce operation further, in order to give a better understanding of the issues that may cause the error:
- When a virtual machine backup jobs starts, a call is made into the vCenter to request a quiesced snapshot of the VM.
- Assuming the VM runs a version of Windows, the vCenter sends a request via the VMware Tools to perform a VSS operation inside the VM, this process being known as a quiesce. Note: If the VM runs a form of Linux then the VMware Tools is still used but instead of VSS, the vmsync driver is used to provide similar functionality to VSS. 
- If the VSS operation is successful this is fed back through the VMware tools to the vCenter that then requests that a snapshot of the VM itself is completed.
- The details of the successful snapshot are then fed back to Commvault so that the backup data can then be streamed
- This quiesce process ensures that the file system inside the VM is in a consistent state for a backup.
- While the backup process in Commvault requests that this process begins (by asking for quiesced snapshots) once the request is made Commvault is not responsible for the success or failure we just await the result, in fact once VMware Tools passes the request to the operating system inside the VM, then a success or fail  is usually a problem within the Operating System
- A failure of the quiesce in effect results in a crash Consistent backup (hence the warning about partial protection against the error and not an outright failure) - this failure automatically triggers a second snapshot request from Commvault that does not request a quiesce
- A restore after a crash consistent backup is similar to the state a server starts up in after interruption of power without a proper shutdown (so most of the time there will be no major issues)
- If the server is running a busy database, then restoration from a crash consistent backup may give a problem which requires support from the database vendor (in the same way that this support may be needed after a power interruption)
- VMware Tools being installed is a requirement for this process and additionally the tools should be up-to-date
- The operating system within the VM must be supported by VMware for quiesce operations.
- It is possible to test the quiesce operation outside of Commvault by using the vSphere Console to request a snapshot with "Snapshot the virtual  machine's memory" unchecked / deselected and "Quiesce guest file system" enabled/selected. If this test is performed and a snapshot is created, remember to remove it after the test. Note: in some cases such a test may need to be done at the time a backup would normally run, as environmental performance differences at different times of day may change the result of the test.
- If VMware Tools is up-to-date and working correctly and the problem persists, then it may be that the VM in question has performance issues or that the processes within the operating system of the VM have a problem. Troubleshooting will then require advice from the operating system vendor and/or VMware and not Commvault. (for a Windows System the Event logs can be reviewed for VSS issues)
 
If VMs exist that either:
- cannot have VMware Tools Installed
- the Operating System does not support a Quiesce operation
- performance or other issues within the VM means that the quiesce operation fails (possibly intermittently)
then there are options within Commvault to workaround the limitations/environmental issues
 
 
In Summary - to troubleshoot  further
1) Make sure the VMware tools are installed and up to date inside all the affected VMs  (list provided below)
2) Run a test of creating a quiesced snapshot in VMware (as described above)
3) Get whoever administers the content of the VMs to confirm what is going on inside the operating system of each VM at the time of the quiesce request (for Windows this means look at VSS issues in the Event logs, for Linux this means researching what the vmsync driver is doing)
4) If there are VMs that will never quiesce then use the following work around options in Commvault.
 
Option 1:
Move the affected VMs to their own Subclient and configure that new subclient for Crash Consistent Backups which will result in the quiesce not being requested.
 
Option 2:
Use the Advanced Client Properties --> Additional Setting called " ignoreQuiesceGuestErrors" which will backup the VMs from the current Subclient Content where those VMs that can quiesce will, and those that cannot will switch to Crash Consistent functionality and not report the warning/error.

Please do not forget to like the answer if the solution worked for you