Crazy Slow Hyper-V Snapshots

Hi ​@Farmer92 ,

Good day!

Try to use the below key and see if it helps

https://documentation.commvault.com/additionalsetting/hypervsnapthreadcount.html

Regards,

Sureshkumar S

Sureshkumar,

   Thank you support did just point out that key this morning as well.   I had seen the key when I was looking for the solution, but the documentation says it is for crash consistent backups.   We are taking quiesced backups… I am going to try the key, but not sure if it will help based on the description.

Thank you,
   

Looks like the description might be correct.  It is only for cash consistent backups.   I did have the additional setting in place last night, and it did not help.

Does anyone else take quiesced IntelliSnap Hyper-V backups?  How log do the take?  I have some that are up to 7 hours… horrible.   In Comparison on VMware the IntelliSnap backups usually took around 20 to 30 minutes.

Hi ​@Farmer92  

When you configure the backup type as "File System and Application Consistent," Commvault requests a production checkpoint from Hyper-V.

Production checkpoints use the guest's VSS (Volume Shadow Copy Service) and Hyper-V Integration Services to quiesce the file system and applications inside the VM, ensuring application-consistent backups.

You can change the backup type to “crash consistent”. A Crash Consistent backup type is exactly as if you attempted a Standard VM Checkpoint within Hyper-V Manager. This change helps workaround the majority of Hyper-V related checkpoint issues. 

Regards,

Kalaivanan. 

Kalaivanan,
  Thank you.  That is something I could test, but most of the time is spent with the multiple discovery's being run against the Hyper-V hosts.  Then those multiple discoveries happening for each of the 50 subclients.  The get queued up behind each other.

   Do you run a Hyper-V shop with IntelliSnap?   How long do your snapshots take? How many hosts in the cluster? How many VMs?

Thank you

Bill

Hi ​@Farmer92  

Hyper-V is highly reliant on WMI (Windows Management Instrumentation) technology. It is used to access information for the Cluster, Nodes and VMs. When WMI query times degrade, this can have a significant impact on our discovery times.

Slow WMI query times can often be found in environments where:

• There are hundreds of VMs running across a small number of nodes.

• Hyper-V is usually being managed by System Center Virtual Machine Manager (SCVMM)

• There are third-party utilities or management packs querying Hyper-V for data (e.g. Veeam Management Pack for SCOM)

Optimizing subclient configuration and WMI performance can significantly reduce overall backup window. Can you try to stagger the schedules as well? 

Regards,

Kalaivanan. 

Kalaivanan,
 Thank you for the great response.

  Do you run a Hyper-V environment with IntelliSnap?  What is the average time frame for your IntelliSnap backup snapshots?  What size is the environment?

Our larger cluster (slow backups) is 11 hosts 400+ VMs with 55 volumes.  Each volume has its own Commvault subclient.

   Looking in VSDiscovery.log in the past, each subclient runs 3 discovery steps, each across the 11 hosts.   So that is in the area of 1800 VM discoveries each evening for the backups… 

My smaller HyperV clusters do not take nearly as long
Hosts * Volumes * 3 disc per = Total number of discoveries
11 * 53 * 3 = 1,749    → Average IntelliSnap time 4:28
6 * 30 * 3 = 540    → Average IntelliSnap time 0:54
7 * 17 * 3 = 357    → Average IntelliSnap time 1:19
4 * 8 * 3 = 96       → Average IntelliSnap time 0:41
3 * 4 * 3 = 36       → Average IntelliSnap time 0:24

Thank you

Bill

Hi ​@Farmer92 

Thanks for the response. To better understand and troubleshoot the issue, we will need to review the relevant logs. Could you please open a support ticket so that we can properly track the issue, investigate it in detail, and work towards a resolution?

Regards,

Kalaivanan. 

Kalaivanan,
   I am sorry I did not realize you were a Commvault employee.  The rintent for my post was to get timings from other customers.

   I had already opened a ticket and have a CMR from back in April.   I have also been pressing on my Customer Success Manger.   In April I was told that this was a known issue and that they were working on it, and we could expect some thing in June/July time frame.   We are nearly at the end of July, and the impression I am getting from my account team is that there has not been any progress yet.   :-(

Thnak you