Solved

browsing Vcenter is take a long time.


Userlevel 1
Badge +9

Hello, 

i have a vcenter which contains a massive  number of vms, each vm is in an individual subclient.

when i try to browse the vcenter it takes more than 10 to 15 mins before viewing  the subclient.

is there anything to enhance the performance of viewing the subclient ? 

if i separated the subclients into multiple backupsets will enhance the listing ? 

if i separated the subclients into multiple backupsets will it affect the retention/dataaging process ?

thanks in advance.

 

icon

Best answer by Onno van den Berg 15 August 2022, 09:58

View original

11 replies

Userlevel 6
Badge +15

Good morning.  Do you know if you are using Subclient or Backupset level indexing for VSA?

Userlevel 1
Badge +9

Hello @Orazan 

 

i can see the only option for indexing is on the vcenter level and its not selected.

appreciate enlighten me if you mean something elese.

 

Userlevel 1
Badge +3

a possible option is to perform a Refresh Datacenters

 

 

https://documentation.commvault.com/11.24/expert/62283_refresh_datacenters.html

 

I few things that could also cause some delays is the proxy being used to perform the discovery.

If you are using a group under the instance level/ Proxy/Access Nodes, I would suggest maybe trying to use a single proxy to if by any chance the group is selecting a proxy outside the network.

 

do you also experience the same slowness if the browse is performed using the command center?

Userlevel 1
Badge +9

@Urosa 

i use only 1 proxy. 

also i have done the datacenter refresh multiple times and i got the same result.

on command center browsing is much better than the commcell console but unfortunately, they removed the option to add the vm/subclient to a storage and schedule policy from command center, you have to use plans. 

Userlevel 1
Badge +9

Also, im trying to restore 4tb of files from a vm to another server and it been 6 hours and only 2gb of 4 TB were restored! 

the restore is sooooooo slow, i dont know why, is there anyway to speed it up ?

 

 

Userlevel 7
Badge +16

Some things here seem to get mixed up or am I misinterpreting this thread?

 

 

 

The initial issue is about a browse regarding already performed backups correct?

 

 

 

The second issue regarding restore can have multiple causes, some initial questions.

 

 

 

First question is, have you encountered this performance issue before or is it the first time encountering this issue? If the first time, have you recently made changes to the environment/configuration?

 

You say you are restoring 4TB of files, are you actually restoring files or a VM? If only files, redirected via an agent or directly into a VM via the hypervisor?

 

 

 

If you are trying to restore it as a VM, some additional questions:

 

You have one proxy, I assume you are referring to the virtual server agent (VSA)?

 

How many CPU and memory does this VSA have?

 

Is the VSA a VM or a physical server?

 

If it is a VM, is it running on the same hypervisor host as where you are restoring to? If not, does the hypervisor where your VSA resides have access to the data store where you are restoring to?

 

Do you use NBD, Hot Add or SAN method?

 

Are there backups running while restoring? Either using the VSA or the same backup target location.

 

How many media agents are involved for that storage policy?

 

What is the network bandwidth available for VSA and Media Agent?

 

What type of backup target is being used? I assume a disklibrary, if so what type of connection. UNC or direct attached or via FC?

Userlevel 1
Badge +9

Some things here seem to get mixed up or am I misinterpreting this thread?

 

The initial issue is about a browse regarding already performed backups correct?

 

The second issue regarding restore can have multiple causes, some initial questions.

 

First question is, have you encountered this performance issue before or is it the first time encountering this issue? If the first time, have you recently made changes to the environment/configuration?

You say you are restoring 4TB of files, are you actually restoring files or a VM? If only files, redirected via an agent or directly into a VM via the hypervisor?

 

If you are trying to restore it as a VM, some additional questions:

You have one proxy, I assume you are referring to the virtual server agent (VSA)?

How many CPU and memory does this VSA have?

Is the VSA a VM or a physical server?

If it is a VM, is it running on the same hypervisor host as where you are restoring to? If not, does the hypervisor where your VSA resides have access to the data store where you are restoring to?

Do you use NBD, Hot Add or SAN method?

Are there backups running while restoring? Either using the VSA or the same backup target location.

How many media agents are involved for that storage policy?

What is the network bandwidth available for VSA and Media Agent?

What type of backup target is being used? I assume a disklibrary, if so what type of connection. UNC or direct attached or via FC?

Hello @Jos Meijer 

yes, i have mentioned 2 issue.

 

  1. slow browsing on vcenter subclients.
  2. very slow restore

regarding the slow restore, im performing a vm backup over SAN, vm size is 5 TB.

im trying to restore a 4tb “guest Files” 

method No.1 i followed is restoring guest files to another vm and it was very slow “restored 12gb im 38 hours”

method No.2 i have tried is restoring guest files to a filesysten agent “server with file system agent installed” and also the progress is very slow “2gb restored in 6 hours”

 

Do you use NBD, Hot Add or SAN method?

SAN

Are there backups running while restoring? Either using the VSA or the same backup target location.

No

How many media agents are involved for that storage policy?

one

What is the network bandwidth available for VSA and Media Agent?

no limits the Media agent has 40Gb interface

What type of backup target is being used? I assume a disklibrary, if so what type of connection. UNC or direct attached or via FC?

FC

Userlevel 7
Badge +16

Regarding the slow browse:
At first glance, there seems to be a general issue here, either in the existing index on the media agent or if the index is not available anymore a disklibrary performance issue while restoring it to the media agent.
But in order to make that conclusion:

  • Is the browse slow loading the VM overview or the file system overview within the VM?
  • On which level are you initiating the browse? On a specific subclient or on a higher level?
    Looking at the print screen, I assume on subclient level?
    If on a higher level, what happens when you initiate the browse on a specific subclient?

 

Regarding the slow restore:
It is inadvisable to per perform guest file recovery via agentless method when looking at such amounts.

Quote documentation “You can use agentless file recovery when the total restore size is less than 10 GB and you are restoring fewer than 10,000 files.”

 

When looking at a redirect via an agent it indeed is a very slow performance, do you get normal performance when restoring a VM or a VMDK? Is this via SAN, NBD or via a separate VSA proxy?

 

In regards of the redirected restore via agent, what source and targets are we talking about. Windows or Linux?

 

Is every component involved on commvault level 11.25.44? As you recently updated the environment.

 

Can you share the restore log for this job? The job for restore via an agent to be specific. This might give some insight.

 

P.s. If the VMDK restore works with normal performance I would mitigate the current situation by restoring the VMDK and mounting it to the target VM and copy data in-guest to the target location. Afterwards you can unmount and remove the VMDK again. Then you wont have as much time pressure to figure this out.

Userlevel 7
Badge +19

Just out-of-curiosity but how many subclients do you have, because you stated each VM is a separate subclient. What was the reason to configure it like this? 

In addition as @Jos Meijer already stated file-level recovery by cracking open a VM without an FS agent creating the backup is very limited when it comes to the volume and recovery speed and thus only intended to perform quick recovery of a small set of files. You should use the FS agent for large datasets!

Userlevel 1
Badge +9

Hello @Onno van den Berg 

 

we have around the 1300 vms in individual subclient. 

we are dividing the vms to simplifying the find and restoring process and the ease of management and reporting.

i have switched the  the large vm backup to agent base backup because we couldn't successfully done the restore of the 4 TB. 

 

Just out-of-curiosity but how many subclients do you have, because you stated each VM is a separate subclient. What was the reason to configure it like this? 

In addition as @Jos Meijer already stated file-level recovery by cracking open a VM without an FS agent creating the backup is very limited when it comes to the volume and recovery speed and thus only intended to perform quick recovery of a small set of files. You should use the FS agent for large datasets!

 

 

Userlevel 7
Badge +19

@Muhammad Abdullah so to remove all the fuzz you have created ~1300 subclients in order to simplify the restore process and reporting? are you only using the CommCell console or also Command Center? I really think you might be hitting an issue due to the high amount of subclients. 

if you really need to restore a large data set from a VSA level backup than it might be faster to recover the entire VM out-of-place and start a copy of the files from the recovered VM, however this entails a lot of manual work and it's not the most efficient route. make sure now you switched to FS-based agent backup to look at the amount of stream to be used during backup because this will improve RTO.

Reply