Restore via SAN problems

Hi Thomas,

If the VM’s Storage is presented to the MA/VSA Read-Only then we will be able to use SAN Transport to backup the VM but not restore.

Have you checked if the storage volume is presented R/W? - You might need to clear the read only flag.

Best Regards,

Michael 

Hello @MichaelCapon , 

I have checked again and can confirm that all storage volumes are presented to the media agents.
How can I just tell all volumes again that they are RW ?
Is there any official documentation from Commvault here ?

with kind regards

Thomas

Hi @thomas.S,

What is the OS of the MA/VSA you are using for Backup & Restores?
Also, what’s the Hypervisor here? I am presuming VMware?

Best Regards,

Michael

Hello @MichaelCapon, 

the Media Agenten are running on Windows Server 2012, Hypervisor is VMware and the VMs are using LUNs from an Huawei Storage. These LUNs are presented to towards the vmware environment and also to the Commvault Media Agents, so we can backup by using transport mode SAN. 

kind regards

Thomas

Thanks Thomas,

From the Commvault Base Directory on the VSA you can run the VMFSSanInfo utility (via Command Prompt) with the “Datastores” or “Disks” switch to print out the VMFS Volume to Disk Mapping.

Ref: https://documentation.commvault.com/commvault/v11/article?p=113660.htm

Example Command: 

C:\Program Files\Commvault\ContentStore\Base\VMFSSanInfo.exe disks

Once you have the output of that, Then you can pick the disk in DiskPart and check the details for: “read-only : yes”.

Select Disk x
Detail Disk

Before making any changes, Ensure Automount is Disabled:

Automount Disable

To clear the read-only attribute for the disk you can run the below:

Select Disk x
attribute disk clear readonly

Let me know if this helps.

Best Regards,

Michael

Hello @MichaelCapon, 

I had time to try this today and executed the command. Is there also the possibility that only disks are displayed which have a read only ?

Kind Regards

Thomas

Hi @thomas.S 

VMFSSanInfo utlitity will display all disks detected with VMFS partitions.

Once you have the DS Name, you can match it with the Disk ID. - Then check the attributes of that Disk.

Best Regards,

Michael

Hi @MichaelCapon ,

We will check this next Monday and perform recovery tests. I will get back to you after that and report the result. 

Kind Regards

Thomas

Hello,

we have not yet been able to perform the tests. We will finally perform them next Tuesday.

Hello @MichaelCapon, 

we have now started a test. Immediately after starting the restore process, the transport mode was set to NBD by Commvault.  We then looked again at the zoning, mapping and read only flag of the LUN. Everything looks correct and we don't know why the restore is not possible via SAN. 
Here are the screenshots.

Kind Regards

Thomas

 

Thanks @thomas.S

Does VMFSSanInfo Util show the DS Name in the output of “disks” or “datastores” command?

I’d suggest checking the vsbkp.log and vixdisklib.log log files on this MA/Proxy for further details here.

You can use GxTail.exe Utility in the Commvault Base directory to review these with filters (F5) and Markers (F9). Look for word strings like “attaching disk” and “transport”.

Let us know what you find.

If you get stuck then I’d suggest opening a case and we can take a deeper look.

Best Regards,

Michael

Hello @MichaelCapon, 

I just opened a case and the solution was found very quickly because we are using Commvault 11.20.90 and VMware 7.0.3 :-) 

I recommend updating to 11.24 so you can get the newer VDDK support. This will resolve your issue from what I see.

There is another possibility as its not clear if the job is failing if you pick SAN for restore.

https://documentation.commvault.com/11.24/expert/32040_san_transport_for_vmware.html

• SAN restores using thin disk provisioning can be slower than LAN restores; performance can be improved by using NBD or by setting the transport mode to SAN and forcing the disk type to thick, which uses eager zero provisioning.

If your VM is thin provisioned Commvault may automatically switch to NBD for better restore performance. So I think a distinction needs to be made between SAN restores being capable vs commvault auto-picking the best transport for the restore at hand. You can check the former by select SAN exclusively on the restore and see if it works. If it does, you are likely falling into this scenario where NBD would actually be faster and hence picked that as the transport method.

Thanks for the update @thomas.S ,
 

I went through your case and the logs here, I can see the error is coming from the VDDK 7.0.0 which is a known issue.

VDDK 7.0.0 was bundled into SP20, VDDK 7.0.1 (which resolved the issue) was included from 11.23 onwards.

VDDK 7.0.1 Release Notes: https://vdc-download.vmware.com/vmwb-repository/dcr-public/91cc8936-cbd5-4f61-9a24-6b9b9e8cd63a/07ff0eec-8401-485e-bab3-5ca4db49905f/VDDK-701-ReleaseNotes.html

You could possibly try dropping the VDDK Back to 6.5.3 and checking the results here. (VDDK supports one release upwards) 
- Below additional settings for the VSA Proxy to do this:

Name: nVDDKVersionPath
Category: VirtualServer
Type: Integer
Value: 673

Name: bLockVDDKVersionPath
Category: VirtualServer
Type: boolean
Value: true

Or you could even try manually installing VDDK 7.0.1 as per the steps here: https://documentation.commvault.com/commvault/v11_sp20/article?p=32585.htm#o127054

Best Regards,

Michael

Hello @MichaelCapon, 

thank you for the information. 
I have made the settings in the media agents and a restore via SAN is now possible. 
Now we have at least gained some time and do not have to immediately start planning an update.
The current restore is running at 350 - 400GB/hr. What else could I check now to get more throughput ?

@Damian Andre: yes, we used thin disk provisioning. Specifying the transport mode had no effect. The job then simply aborted. 

Thanks for the update @thomas.S ,

With regards to performance, you can check the “vsrst.log” and the “CVD.log” in the Log Files directory on the machine performing the restore (VSA Proxy/MA).

You can use GxTail.exe Utility in the Commvault Base directory to review these with filters (F5) and Markers (F9). Filter these logs on the JOBID of the restore and look for strings like “stat-” or “speed”, which should show the statistics for the modules.

-Hopefully you should be able to see the lower statistic, which would indicate where the bottleneck might be.

Let me know what you find.

Best Regards,

Michael

Hi @MichaelCapon, 

OK, thanks for the information, I will have a look at the logs tomorrow. 

Kind Regards

Thomas

@thomas.S , can you share their findings for posterity?

Thanks!