Solved

Granular Recovery issue from Azure Library

  • 2 November 2022
  • 7 replies
  • 141 views

Badge +4

 Hello Experts,

 

Firstly, please accept my apologies for my lengthy post. 

 

I need your assistance with a problem we're having while performing granular data recovery of VM data from Azure Library.

 

We have VSA backups with extended retention in an Azure cloud library (hot+Archive tier) that are shared by two Windows media agents.

 

Attempting to recover data from a File Server instance.This file server has multiple virtual drives (vollumes), with three drives (C: E: H: each drive less than 1 TB) on one datastore and two more on another datastore (S: V: - with S: volume of 10 TB and V: volume of 5 TB). This VM/Subclient is backed up with intellisnap, and the backup is then copied to Azure.
When attempting to browse through the "Files and Folders" option from the Azure backup, the VSA subclient fails with the error " Browse failed due to error while mounting virtual machine."
I have already completed a recall index and Recall Archive storage workflow with success.

 

I thought the pseudomount was working because I could browse through a few volumes.
However, I later discovered that the S: and V: drives were not showing any data and were throwing pseudo mount errors.
I can drill down and see all of the volumes (as shown in photo 1), but I can't drill down any further for volumes S: and V: ; we are seeing the errors below with the live browse/pseudomount.
The pseudomount logs show repeated failures.
We also tried increasing the "SRV CLIENT RCV TIMEOUT" to 36000, but the problem persisted.


Could this be a problem with WAN latency (it's only 400ms)?


Should we copy the entire job to local disc and then attempt granular recovery via live browse?
What are your recommendations? Greatly appreciate your advise to fix this problem.

 

2022-11-01\ma02\PSEUDOMOUNT.log:
2932  17bc  11/01 10:17:48 ###### CPostRecallHandlerKernel::Done(1387) - One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774), Failed to complete IO request for ReqId=[1973] Lun=[0:0:2] Extents=[6203167-6203167] isRead=[1] (W32.774): 0x80070306:{CPostRecallHandlerKernel::Done(1387)/0x80070306:{CKernelIoHandler::SendIoResponseToKernel(1054)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Failed to complete IO request for ReqId=[1973] Lun=[0:0:2] Extents=[6203167-6203167] isRead=[1]}} + {CKernelIoHandler::SendIoResponseToKernel(1054)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Failed to complete IO request for ReqId=[1973] Lun=[0:0:2] Extents=[6203167-6203167] isRead=[1]}
 
. . .
2932  17bc  11/01 10:17:48 ###### CRecallContext::RecallComplete(82) - Job completed Status: Status FAIL; JobId=0; Path=503468fa-fea2-964b-aa65-283c0e080341\scsi0-2-.vmdk\0005T9OV

 

So you know, we can browse into the S: volume on newer backups that are on local disc copy and not on Azure extended retention backups.
Also, I'd like to keep the idea of copying the Job ID from Azure to the local disc copy as a last resort because we're talking about close to 5TB of data (Post-compression) and I'd like to avoid egress charges as much as possible. 

 

 

 

icon

Best answer by Mike Struening RETIRED 30 November 2022, 21:23

View original

7 replies

Userlevel 7
Badge +23

No apologies needed, details are always great to have, @Nik !

I see we have an update to address this error in 11.24.21.  Can you confirm your CS and MA version?

If you have a different Feature Release, I can look into the form ID and see which MR fits your FR.

Badge +4


Thanks for your response @Mike Struening  , the CS and MA are on version 11.28.26.  Do you have a bug id that I can share with support? 

Userlevel 7
Badge +23

Thanks for the quick reply, @Nik !

The update was a hotfix for 11.24, though I do not see a version for 11.28.

I would suggest opening a support case and having them confirm this issue is the same as fixed in Form ID 3219 for 11.24 (Support can look this up and escalate accordingly).

Share the case number with me so I can track it on my end.

Badge +4

Sorry for not getting back sooner @Mike Struening . The support case number is 221028-22.

 

It has now been escalated to the CommVault Development team are actively working on a fix for this issue. I will let you know what the proposed steps are once they have tested.

Userlevel 7
Badge +23

Awesome, glad to hear it!

I’d rather the issue be getting fixed and I find out later rather than the opposite 🤓

Badge +4

The Commvault development team investigated the logs and reproduced the problem in their own labs.

They were able to find an issue where if the corresponding ". SIZE" file for a data chunk was located in HOT tier, the restore attempted to read from the regular container rather than the cvshadowcopy container . I was told that starting with SP28, CV will try to recall/browse recalled data from the cvshadowcopy location rather than the 'live' container.

 

CV created a hotfix for this, which was applied to our Media agents, and the cloud restores functioned normally. This fix, I was told, will be included in future releases.

 

Thanks for your help overall!

Userlevel 7
Badge +23

For anyone else following, it’s Form 3368 in 11.28.36.

If anyone needs the release for another Feature Release, let me know!

Reply