Skip to main content

Hi all,

After changing the Media Agent (MA), we are experiencing very slow VMware backup speeds, approximately 1MB/s using NBD mode. During troubleshooting, the only noticeable issue was a high 99% read rate. Other than that, nothing particularly helpful has been identified.

 

Any suggestions for further investigation?

Hi @drPhil,

Is the MA being used as the VSA Proxy here? and are there any other transport modes that can be leverages? (SAN if Physical LUN access is present on the MA/VSA or HotAdd if the VSA is a VM in the same VMware environment)

You can look at the vsbkp.log logfile on the VSA Proxy and look at the “stat-” counters to see the read (readdisk) and write (WritePLBuffer) speeds on the VSA side of the Backup. - The lower value is going to be the bottleneck.
For the MA side of the backup, the cvd.log logfile will contain the write speeds to the media.
We also have the CvPerfMgr.log on the MA side which details each segment of the process and their speeds. - This may also help identifying bottlenecks.

If the NBD speed is the bottleneck here, I’d suggest checking the performance of the NW and ensure that the hostname of the ESX is resolving to the appropriate NW interface of the ESX.


If this is a more urgent matter, then I’d suggest opening up a support case so that we can check this further for you.

Let us know how you get on.

 

Best Regards,

Michael


Hi @MichaelCapon, indeed, the NBD mode (physical MA as VSA proxy) seems to be the bottleneck, but in a very strange way. Within the backup set, most of the VMs are backed up at the usual speed, but the last two or three VMs are backed up at a speed of only 2MB/s. There is nothing unusual (e.g., CPU usage) with the affected VMs. In the Commvault logs, we noticed that most of the time is taken up by disk read speed. Has anyone experienced a similar issue?


Thanks @drPhil,

Are these VM’s on a specific ESX or Datastore? Are any other VM’s using the same Host/Storage?

If you move the VM to another Host/Datastore, does the issue continue here?

Are there many Snapshots on those VM’s or VM Datastores also?

 

Best Regards,

Michael


Agree with Michael.  If we’re seeing 99% on reads, my first step would definitely see if the VMs can be moved to a different DataStore to see if the issue follows them.

Thanks,
Scott
 


Big thanks to @MichaelCapon and @Scott Moseman  for the suggestions. Unfortunately, we couldn't dive deeper into the slow NBD backup issue due to a lack of VMware experts and time. However, we installed a VSA proxy and switched to Hotadd mode, which significantly improved VMware backup performance (up to 450MB/s for some subclients). That said, performance for other subclients wasn’t as impressive, leading us to consider using multiple VSA proxies for load balancing. Overall, I recommend rethinking the backup strategy and opting for more efficient transport modes, rather than focusing too much on troubleshooting the virtualization platform itself.


Reply