Skip to main content
Solved

VMware backup extremely slow

  • 24 September 2024
  • 5 replies
  • 179 views

Forum|alt.badge.img+15

Hi all,

After changing the Media Agent (MA), we are experiencing very slow VMware backup speeds, approximately 1MB/s using NBD mode. During troubleshooting, the only noticeable issue was a high 99% read rate. Other than that, nothing particularly helpful has been identified.

 

Any suggestions for further investigation?

Best answer by drPhil

Big thanks to @MichaelCapon and @Scott Moseman  for the suggestions. Unfortunately, we couldn't dive deeper into the slow NBD backup issue due to a lack of VMware experts and time. However, we installed a VSA proxy and switched to Hotadd mode, which significantly improved VMware backup performance (up to 450MB/s for some subclients). That said, performance for other subclients wasn’t as impressive, leading us to consider using multiple VSA proxies for load balancing. Overall, I recommend rethinking the backup strategy and opting for more efficient transport modes, rather than focusing too much on troubleshooting the virtualization platform itself.

View original
Did this answer your question?

5 replies

MichaelCapon
Vaulter
Forum|alt.badge.img+14
  • Vaulter
  • 349 replies
  • September 24, 2024

Hi @drPhil,

Is the MA being used as the VSA Proxy here? and are there any other transport modes that can be leverages? (SAN if Physical LUN access is present on the MA/VSA or HotAdd if the VSA is a VM in the same VMware environment)

You can look at the vsbkp.log logfile on the VSA Proxy and look at the “stat-” counters to see the read (readdisk) and write (WritePLBuffer) speeds on the VSA side of the Backup. - The lower value is going to be the bottleneck.
For the MA side of the backup, the cvd.log logfile will contain the write speeds to the media.
We also have the CvPerfMgr.log on the MA side which details each segment of the process and their speeds. - This may also help identifying bottlenecks.

If the NBD speed is the bottleneck here, I’d suggest checking the performance of the NW and ensure that the hostname of the ESX is resolving to the appropriate NW interface of the ESX.


If this is a more urgent matter, then I’d suggest opening up a support case so that we can check this further for you.

Let us know how you get on.

 

Best Regards,

Michael


Forum|alt.badge.img+15
  • Author
  • Byte
  • 110 replies
  • September 25, 2024

Hi @MichaelCapon, indeed, the NBD mode (physical MA as VSA proxy) seems to be the bottleneck, but in a very strange way. Within the backup set, most of the VMs are backed up at the usual speed, but the last two or three VMs are backed up at a speed of only 2MB/s. There is nothing unusual (e.g., CPU usage) with the affected VMs. In the Commvault logs, we noticed that most of the time is taken up by disk read speed. Has anyone experienced a similar issue?


MichaelCapon
Vaulter
Forum|alt.badge.img+14
  • Vaulter
  • 349 replies
  • September 25, 2024

Thanks @drPhil,

Are these VM’s on a specific ESX or Datastore? Are any other VM’s using the same Host/Storage?

If you move the VM to another Host/Datastore, does the issue continue here?

Are there many Snapshots on those VM’s or VM Datastores also?

 

Best Regards,

Michael


Scott Moseman
Vaulter
Forum|alt.badge.img+18

Agree with Michael.  If we’re seeing 99% on reads, my first step would definitely see if the VMs can be moved to a different DataStore to see if the issue follows them.

Thanks,
Scott
 


Forum|alt.badge.img+15
  • Author
  • Byte
  • 110 replies
  • Answer
  • October 9, 2024

Big thanks to @MichaelCapon and @Scott Moseman  for the suggestions. Unfortunately, we couldn't dive deeper into the slow NBD backup issue due to a lack of VMware experts and time. However, we installed a VSA proxy and switched to Hotadd mode, which significantly improved VMware backup performance (up to 450MB/s for some subclients). That said, performance for other subclients wasn’t as impressive, leading us to consider using multiple VSA proxies for load balancing. Overall, I recommend rethinking the backup strategy and opting for more efficient transport modes, rather than focusing too much on troubleshooting the virtualization platform itself.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings