What is the best method to restore files from a guest VM protected by VMware backups?

Hi Christine, 

what i have seen in the field, a restore of guest files direct in a vm is sometimes slower than a restore to an agent and based on the feedback from Jordan there are a lot of options to restore files. 

try it on your environment to get a feeling for it!

Hope that helps you a little bit

Cheers Philipp 

Totally agree and my experience as well. There are protocol slowdowns by routing data through the ESXi webserver which is required for such restores. Almost always you can replicate this by using powercli and running the Copy-VMGuestFile function. This is essentially the same function that Commvault is calling, so its a good way to troubleshoot or validate the bottleneck with slow file restores directly to the guest VM. But lucky that there are multiple options to restore guest files, so if you have larger restores you can use another method!

It’s been discussed that the MS Windows MA’s running FR11.22 have a significant performance enhancement such that having an in-guest agent may no longer be required. @Damian Andre were you able to get some technical documentation about these improvements? 

Hello @Anthony.Hodges ,

One of the biggest bottlenecks when restoring guest files using the agentless method (uploading files directly to vCenter, and using VMTools to inject files into a guest VM), is that the copy-VMGuestFile API is single threaded, and requires a response from vCenter to confirm that the file has in fact been restored correctly.

This may not cause you much bottleneck with a small amount of files, but the more files restored, the more impact you will notice on the performance. 

Alternatively, using an in-guest agent to restore, we can restore files using a multithreaded approach. As such, you will notice a large performance improvement when restoring a large amount of files.

Not aware of any changes. I think there is some functionality change for AHV but not for VMware. Seems unlikely unless there is a new VMware API to consume - the functionality on the Commvault side is straightforward. I’ll ask around!

@Anthony.Hodges - Just got word - the enhancements in FR22 are focused around live-browse, so read performance improvements for all recovery methods rather than specifically improving the write performance of agentless recovery. However, they did let the cat out of the bag that they are working on some secret voodoo that should improve agentless recovery. I tried to dig in more but …
 

Look at it depending on how big the VM is ( I think) the best approach. If you have a 80gb VM you need files off of its easy to restore the vm files or sometimes the live vm. IF you have a 10tb VM, its easier and faster to restore the drive to another vm and just transfer it. Again this is simply suggestion and my opinion. I have found this to be a good rule of thumb when restoring files from a vm.

Damian Andre wrote:

Damian Andre wrote:

Anthony.Hodges wrote:

It’s been discussed that the MS Windows MA’s running FR11.22 have a significant performance enhancement such that having an in-guest agent may no longer be required. @Damian Andre were you able to get some technical documentation about these improvements? 

Not aware of any changes. I think there is some functionality change for AHV but not for VMware. Seems unlikely unless there is a new VMware API to consume - the functionality on the Commvault side is straightforward. I’ll ask around!

@Anthony.Hodges - Just got word - the enhancements in FR22 are focused around live-browse, so read performance improvements for all recovery methods rather than specifically improving the write performance of agentless recovery. However, they did let the cat out of the bag that they are working on some secret voodoo that should improve agentless recovery. I tried to dig in more but …
 

 

Thats good to know. 

Same over here. We have a vm with million of files. Individual restore of files was impossible (using the VM backup). With a copy & transfer of the files it did work.

Hello Christine, As other community members comment, there are many options for restoring individual files to a virtual machine. But if the restoration is of many files or of a very large volume of data, one option would be to restore the entire disk at the VMDK level, attach the disk to the original machine and copy / paste the files at the operating system level, from the new disk to the required path. If the client machine is Windows, this option is easier to do.

if you know how many files\folders you want to restore then you have different option.

1. if very few files\folders then agentless restore is good option.
2. if you have large file\folders then agent based restore is good option. you just need to have commvault tools installed on the destination VM (File system core package).
3. if you very very large files\folders restore then you can  always do VMDK file level restore or attach disk to VM functionality.

let me know if you have any further questions\concerns.

A file level restore from a Virtual Server backup will restore from block to file level. Since the backup does not backup the file but the blocks they are included in, the restore brings the block back and then pull the files from the blocks. If the file is spread out across 10 blocks., the restore has to first grab each one of those blocks out of the backed up chunks. It then needs to restore each block into a holding location and once it has all the blocks it can reconstruct the file and then place it to the restore location. 

1) Perform a live browse of the data as this will use less space on the proxy building the data, but will still have performance issues for large restores.
2) Restore the VMDK the out place and attach to a VM. Copy the files you need then discard the disk.
3) Choose less files to restore at the same time.

Even if you were looking to restore just 1 file, it maybe spread across 10 blocks. The restore has to first grab each one of those blocks out of the backed up chunks and then restore each block into a holding location. Once it has all the blocks it can reconstruct the file and then place it to the restore location.
 
There are different options for these file level restores as listed here: http://documentation.commvault.com/commvault/v11/article?p=products/vs_vmware/c_vmw_restore_guest_files.htm

 
 
A 'guest agent' restore uses a Windows File System iDA installed on the client to run the restore. This is the most reliable and robust option for the restore of the data. 

https://documentation.commvault.com/commvault/v11_sp20/article?p=36492.htm

Note: For the fastest recovery time and the least space required for extent caching, use Live File Recovery with a restore-only file system agent in the guest.

You can use agentless file recovery when the total restore size is less than 10 GB and you are restoring fewer than 10,000 files.

To restore virtual machine files and folders when the total restore size is larger than 10 GB or has more than 10,000 files, use a restore-only file system agent installed in the destination client or virtual machine.
 
An agentless restore has performance issues when restoring files at 10GB and\or 10k files. There are also additional caveats such as inability to restore ACLs.

https://documentation.commvault.com/commvault/v11_sp20/article?p=36497.htm

This looks like what I will need to do for certain servers where I know restores of large files are likely to happen. Thank you so much for the advice.

Hello!

If you need to restore some files from a File Server VM (~3 TB capacity), what is the best way to go?

From Command Center → Virtualization and then “Guest files” restore, I noticed that sometimes have a delay while browsing the contents  (Browsing contents may take some time. Please wait...).

Is this normal or there is another faster way to browse these data and restore?

Thank you in advance!

@Nikos.Kyrm I believe the delay is normal when there are a lot of files or a lot of large files. To build the list of files and display the directory structure in the GUI, the backed up VMDKs are being virtually mounted in the background which requires a lot of I/O hence the delay. You have the option to force backed up files to be indexed immediately after backup jobs take place, but that requires time and CPU cycles. I believe the index option is disabled by default -- the reasoning may be that restores happen far less frequently than backups.

It seems to be a “pay me now or pay me later, but you’re going to pay” type of situation.

A file level restore from a Virtual Server backup will restore from block to file level. Since the backup does not backup the file but the blocks they are included in, the restore brings the block back and then pull the files from the blocks. If the file is spread out across 10 blocks., the restore has to first grab each one of those blocks out of the backed-up chunks. It then needs to restore each block into a holding location and once it has all the blocks it can reconstruct the file and then place it to the restore location. 

1) Perform a live browse of the data as this will use less space on the proxy building the data but will still have performance issues for large restores.
2) Restore the VMDK the out place and attach to a VM. Copy the files you need then discard the disk.
3) Choose less files to restore at the same time.

Even if you were looking to restore just 1 file, it may be spread across 10 blocks. The restore has to first grab each one of those blocks out of the backed-up chunks and then restore each block into a holding location. Once it has all the blocks it can reconstruct the file and then place it to the restore location.
 
There are different options for these file level restores as listed here: http://documentation.commvault.com/commvault/v11/article?p=products/vs_vmware/c_vmw_restore_guest_files.htm

 
A 'guest agent' restore uses a Windows File System iDA installed on the client to run the restore. This is the most reliable and robust option for the restore of the data. https://documentation.commvault.com/commvault/v11_sp20/article?p=36492.htm

Note: For the fastest recovery time and the least space required for extent caching, use Live File Recovery with a restore-only file system agent in the guest.
You can use agentless file recovery when the total restore size is less than 10 GB and you are restoring fewer than 10,000 files.
To restore virtual machine files and folders when the total restore size is larger than 10 GB or has more than 10,000 files, use a restore-only file system agent installed in the destination client or virtual machine.
 
An agentless restore has performance issues when restoring files at 10GB and\or 10k files. There are also additional caveats such as inability to restore ACLs. https://documentation.commvault.com/commvault/v11_sp20/article?p=36497.htm

Hello Ron and thanks for your reply!

Where is this option about indexing (for file level restore)?

In case of this File Server maybe is useful !

Hi @Nikos.Kyrm 

It’s in the properties of your VM subclient that you would set you want indexing to be performed, post backup : 

Backup options tab, and at the bottom, checkbox to Collect file details

https://documentation.commvault.com/11.25/expert/62285_subclient_properties_backup_options.html

I think in practice this is a really hard question to answer, due to practical constraints. 

I will say that they best solution I have seen personally is a customer that provisioned a  large restore vm in each datacenter to correspond to their physical location. This client was running the Filesystem agent and acting as a media agent. This made things like transport type irrelevant. 

You could restore entire vmdks from san ndb etc with the same ease. Things like networking constraints are already address by virtue of being part of the cluster that was being backed up.

if this is an option for you, it can significantly reduce the headaches associated with restores.