VMware SAN transfer backups about 25% slower after 11.36 upgrade

Hi ​@Farmer92 ,

Good day!

Need VDDKs are added in SP36, From vsbkp.log verify what VDDK is being used now to make sure its not the issue.

Also check if there is any Version upgrade on VC/ESXI end

From old and new jobs logs or history we need to verify which part is the bottleneck where is are spending most time.

Regards,

Sureshkumar S

Sureshkumar,

   Thank you.  I did look and the logs VixDiskLib shows VDDK702 before and after the 11.36 upgrade.

I am trying to get support to help find out what changed between 11.28 and 11.36 that may slow things down.

   I also see that at least one of my AUX copies, from our main datacenter to a remote data center also slowed down since upgrading to 11.36 on Jan 9th.   The blue transfers will not transmit as much as they did prior.

   Something is not quite right with 11.36

I am seeing the same issues with the upgrade from 11.32 to 11.36 and have an open ticket with CommVault 250317-573. They are recommending I check for bottlenecks in the network which we have ruled out as being the issue. We were also given a manual backup via command line to isolate the backup from software but I feel like all the same settings would be used. C:\Program Files\Commvault\ContentStore\Base> .\TestVMInfo.exe -vmware -server 'your vcenter fqdn' -user 'username' -password 'password' -vmname 'YourVM' -desthost 'host fqdn' -destDatastore 'datastore name'  -transport SAN -disksize 73 -readdisks But this has not given us any additional information

Results came back and I ran it using both SAN and NBD and NBD completed but SAN is still running so I feel like it is tied to backups running over SAN mode. Did anyone get this resolved?

Hi Greg,

Kindly compare the results as below :

1. Determine the Jobid of the backup performed via SAN mode and look into vsbkp.log , VixDiskLib.log  on the VSA proxy and CVPerfmgr.log, CvPerfLogAnalyze.log on the MA 
2. Determine the Jobid of the backup performed via NBD mode look into vsbkp.log , VixDiskLib.log  on the VSA proxy and CVPerfmgr.log, CvPerfLogAnalyze.log on the MA 

Comparison needs to be performed for both the scenario’s in order to determine and find out the bottleneck . CvPerfLogAnalyze.log will provide a summary of the performance and provide some insights about the bottleneck . These results will need to be looked at . As you have also mentioned about Auxcopy performance issue , MA side will need to be looked at (CVPerfmgr.log, CvPerfLogAnalyze.log)

Following link provides the high level overview of the changes with regards to Virtualization in general 

Changes in Commvault Platform Release 2024E  

From the Fabric/HBA stats , there have been slow performance recorded during May & July  time frame as well (Though the performance stats are stable and optimal from Mid-July to Early Jan time frame), hence upgrade to 11.36 may not be the cause of the issue . 

If this needs a further diagnosis , would be best to log a support ticket with the observations and results recorded as per the recommendations above. 

Thanks & Regards 

Hi Greg,

I still have an open support ticket regarding the issue. So far, they suspect it may be related to DDB performance. Since we upgraded to 11.36, the Q&I time has increased, but they haven’t been able to pinpoint any specific reason for the slowdown after the update.

The upgrade was the only change in our environment at the time. While we do have a very large partitioned DDB, and they believe we might be nearing a threshold, it’s unclear why the performance would degrade so significantly right after the upgrade.

The investigation continues.

​@Farmer92 did you have a resolution for this case? do you mind sharing with us the current scenario?

​@dude,
  Unfortunately, we're still working through the ticket and have been engaged with support since January 23rd. This issue continues to impact the performance of all our backups.
   One thing I discovered in the logs is that when we were on version 11.28, the DDB engine was c-TreeACE 11.6. Now, in CV 11.36, the DDB engine has been upgraded to FairCom DB 12.6. This change in the database engine could be a contributing factor, but I’m waiting on further insights from support.
   So far, we’ve attempted DDB re-indexing jobs and made memory adjustments, but neither had a significant impact on backup performance.
   I appreciate you reaching out, and I’m open to any other ideas you might have.

Thanks for sharing ​@Farmer92 I just recently upgraded from 11.32 to 11.36 and seeing some slowness as well. Still dont have a ticket, but will collect some evidence before opening one. Thought I would reach out and see where you were. Please let us know if you hear back from support. Thank you

Yes, please gather info and open a ticket.   CV support says they do not have others with the issue, so they do not have others to compare with.

Not sure if it will help or not, but when you open your ticket you can mention my ticket if you like 250123-84.

Also if you can look at your CSTStatus.FSC file to a time prior to your upgrade I would be curios to see what DDB engine version you had in 11.32.  The file should be in your DDB folder under CV_SIDB\2\<nuber>\Group00.

Thanks

​@dude , 
   Also out of curiosity, would you mind sharing a screenshot o your DDB statistics, similar to below?  I am wondering if there is some size component to the slow down.  The chart is for our top level of our DDB.  We have 4 partitions under it to get to that total number.