Live Browse for Windows VMs failed with error code -6

Hi Kelvin,

I’d suggest checking CVD.log and PSEUDOMOUNT.log on mil-commvault-2 at/before 03/15 15:21:07 for any errors/messages that may be related.
I would also suggest ensuring automount is disabled on the MA machine also.

Kind Regards,

Michael 

Hi Michael,

Thanks for you quick reply.

I just did a quick check but still not sure where to start ?

For example,

Automount is disabled (in fact, it was enabled but then, disabled by the Commvault support)

PSEUDOMOUNT.log shows that it failed to initialize and mount the PSEUDO disk, if I read the below log correctly ?

“

15:21:07 ### CPseudoVMManager::CreateAndMountPseudoDevices(168) - Partial drive listing. Skipping disk since it failed to initialize: device=5006bb64-d4a7-2e4f-a840-178ed2b3f384_41145_scsi0-0-ML-RAS3.vmdk, error=0x80070001:{CVSAPseudoDevice::CreatePseudoDisk(123)/Disk Initialize failed} + {CVSADisk::Initialize(739)} + {CVirtualDiskBase::Initialize(136)/ReadAheadExtents failed} + {CVSADisk::ReadAheadExtents(676)} + {CVirtualDiskBase::ReadAheadExtents(476)/Recall failed!} + {CVSAPseudoDevice::Recall(217)} + {CVSAPseudoDevice::PostRecallTask(293)} + {CPseudoDevice::PostRecallTask(640)} + {CVirtualDiskBase::Recall(519)} + {CVirtualDiskBase::RecallInternal(535)/RecallExtent failed!} + {CVSADisk::RecallDataFromSource(559)} + {CVSADisk::RecallDataFromSource(554)} + {CRecaller::RecallExtent(298)} + {CRecaller::RecallExtent(255)} + {CRecaller::SubmitRecallRequest(378)/W32.1.(Incorrect function.)-Failed to submit restore request to DMRService, error=29}

21472 5af4  03/15 15:21:07 ### CPseudoMountServer::HandleVMMountReq(1394) - --- 0:01.502924 

21472 5af4  03/15 15:21:07 ### CLibCvBlkLevelRestore::IncomingMsgXMLCallback(234) - VSABLKVMMountReq failed, error=0xEDA00003:{CPseudoMountServer::HandleVMMountReq(1553)/MountVM failed, VM=5006bb64-d4a7-2e4f-a840-178ed2b3f384} + {CPseudoVMManager::MountVM(350)/Failed to create/mount pseudo devices} + {CPseudoVMManager::CreateAndMountPseudoDevices(197)} + {CPseudoVMManager::CreateAndMountPseudoDevices(195)/CV.0x3-0x80070001:{CVSAPseudoDevice::CreatePseudoDisk(123)/Disk Initialize failed} + {CVSADisk::Initialize(739)} + {CVirtualDiskBase::Initialize(136)/ReadAheadExtents failed} + {CVSADisk::ReadAheadExtents(676)} + {CVirtualDiskBase::ReadAheadExtents(476)/Recall failed!} + {CVSAPseudoDevice::Recall(217)} + {CVSAPseudoDevice::PostRecallTask(293)} + {CPseudoDevice::PostRecallTask(640)} + {CVirtualDiskBase::Recall(519)} + {CVirtualDiskBase::RecallInternal(535)/RecallExtent failed!} + {CVSADisk::RecallDataFromSource(559)} + {CVSADisk::RecallDataFromSource(554)} + {CRecaller::RecallExtent(298)} + {CRecaller::RecallExtent(255)} + {CRecaller::SubmitRecallRequest(378)/W32.1.(Incorrect function.)-Failed to submit restore request to DMRService, error=29}}

21472 5af4  03/15 15:21:07 ### CLibCvBlkLevelRestore::IncomingMsgXMLCallback(256) - Response=<?xml version="1.0" encoding="UTF-8" standalone="no" ?><FBR_VSABLKMountVMResp Resp="1359" VMGuid="5006bb64-d4a7-2e4f-a840-178ed2b3f384"><err errorCode="3" strError="Failed to intialize VM disks"/></FBR_VSABLKMountVMResp>

“

CVD.log, however, reports “MountVM failure with error code 3”, and “Failed to open XML file()” as below

“

6044  7ee0  03/15 15:21:06 ##### CVSAPseudoMountClient::MountVM(276) - +++ 

6044  7ee0  03/15 15:21:07 ##### CVSAPseudoMountClient::MountVM(290) - Mount Response Received, status=1359 

6044  7ee0  03/15 15:21:07 ##### CVSAPseudoMountClient::MountVM(302) - Mount Response Received, Resp=<?xml version="1.0" encoding="UTF-8" standalone="no" ?><FBR_VSABLKMountVMResp Resp="1359" VMGuid="5006bb64-d4a7-2e4f-a840-178ed2b3f384"><err errorCode="3" strError="Failed to intialize VM disks"/></FBR_VSABLKMountVMResp> 

6044  7ee0  03/15 15:21:07 ##### CVSAPseudoMountClient::MountVM(276) - --- 0:01.506492 

6044  7ee0  03/15 15:21:07 ##### PseudoMountClient MountVM failed with error code [3] error Message [Failed to intialize VM disks]

6044  7ee0  03/15 15:21:07 ##### VMLivebrowse::ReleaseLivebrowseObject() - [VMLivebrowse] Live browse object [0x000000ABBDED8A10] reference count [0]

6044  8c98  03/15 15:21:07 ##### LiveBrowseRemoteMessageHandler() - instanceType 0

6044  8c98  03/15 15:21:07 ##### vsXMLFile::Load() - Failed to open XML file []. Error: [0xECCC000D:{CQiFile::Open(95)} + {CQiUTFOSAPI::open(81)/ErrNo.13.(Permission denied)-Open failed, File=\\?\C:\Windows\system32, OperationFlag=0x8000, PermissionMode=0x100}]

6044  8c98  03/15 15:21:07 ##### VMLivebrowseUtils::GetVMMountInfoFromFile() - [VMLivebrowse] Failed to load XML file []

6044  8c98  03/15 15:21:07 ##### VMLivebrowse::Handle_FileSystemGetItemCount() - [VMLivebrowse] _GetVMMountInfoFromFile failed

”

Hey @Kelvin !  Thanks for the post.  Is there any reason you don’t want to upgrade the other client to 11.20?  11.16 is the lowest supported version, so I can see that Customer Support gave a best effort, though they can only do so much.

I’d certainly like to see if anyone else has any experience, though the upgrade to 11.20 is not a complex process.

Hi Mike,

This is because I’m not too sure how relevant the version is in this case, unless this LiveBrowse has never worked before with 11.14 on our CommServe No.2...

That being said, this might be the case as I’ve just got this job 1 month ago and my predecessor doesn’t seem to be able to recall when was the last time this function still worked LOL

But, even though, as an educated guess, I’d assume that at the time when this CommServe No. 2 was installed, if configured properly, there shouldn’t have been any reason why this function would have not worked ?

So, I guess that I’m just trying to rewind the time back to 4 years ago, and try to find the old way to fix the problem LOL

Fair enough @Kelvin    One thing I would check though is the history of this client and see if this EVER worked.  I have absolutely seen plenty of setups where it never worked, it was pushed to the side, then X years later someone gives it a go and wonders why it stopped working.  Try a Job summary report and see what you find.

I did find your ticket and made an internal note so I could track it, though as you mentioned it is waiting on an update to 11.16 or higher (and since the other one is on 11.20, I’d go that route).

Other incidents I found with your error were resolved by disabling AV (it was locking up binaries) or uninstalling and reinstalling.

If you can find when it worked before, that would help, though keep in mind, Support is looking to send this to development for review, and dev only makes updates for supported features (11.16 is the minimum long term and 11.20 is the higher long term release).

Keep me posted and let me know if I can help in any way!

Hi Mike,

Having tried to completely disable the SentinelOne on the CommServe, as well as to enable the NFS on the NAS that makes up the disk library, the LiveBrowse still doesn’t work with the same error message.

So, we are now going to upgrade the CommServe to SP16 first so as to get the Commvault support back in again.

Therefore, could I ask you the below questions:

1) Our vCentre Server version is at 6.0.0, which is pretty old, is it compatible with SP16 ?

2) Do you know, roughly, how long the upgrade take ? Knowing this would help me to decide whether to do the upgrade on a week day or at a weekend.

3) Is the upgrade itself very simple and straight-forward ? For example, do I need to read a manual/guide of it ?

Sorry for asking those very basic questions, as I'm from Netbackup background, and their upgrade was never as simple as what they said on the tin…

Thanks,

Sorry, Mike, I have one more silly question.

After upgrading to SP16, if the CommServe has some major backup issues, can we roll back to the current SP14 ? Or, we will be rather stuck with the new version ?

No silly questions at all!  this is 100% the perfect place for this.

I’ll answer as you asked:

1. vCenter 6.0.0 is fine for 11.16 or 11.20 (or higher.  Version requirements are here
2. ROUGHLY….maybe 15-20 minutes per server?  That’s a touch one to answer, though you’ll want to do the CS first, then the Media Agents (mostly likely) followed by any clients….and only some phases can be done simultaeously
3. The update process is REALLY straightforward.  Should be quite smooth.  If there *IS* a rare issue, there’s no need to uninstall; just run a repair
4. You can always reinstall on 14 and restore a DB from 14 (I would definitely run a DR backup before you do the upgrade), but I also would expect a smooth process.

I do want to reiterate that 11.16 is the oldest Long Term Release, so I would suggest 11.20 if possible.  If you go to 11.16, you’ll find yourself needing to upgrade before you know it, but 11.20 will future proof you for a long time!

Keep me posted!

Hi Mike,

I’ve downloaded the SP16 and its 1.6GB hotfix, and I think I’m almost ready for the upgrade (I’ll take it as my first experience before feeling confident enough to do it again, i.e. upgrading it to SP20.36).

Our system is quite simple, just one CommServe with 2 dead Media Agents (so I don’t have to worry about them), and very few Clients of no importance.

The key thing, however, is the vCentre client defined in Client Computers.

Having checked the property of this vCentre Client, unlike other Clients, I couldn’t find any option to show its current version.

Does this mean that I don’t need to do anything to this vCentre Client during the upgrade as the upgrade to the CommServe itself should take care of it ? 

Thanks,

Most likely, you are looking at the virtual client, not the Proxy that accesses the vms.  the Proxy should have a version.

If your environment is small, just update all clients once the CS, then MAs are updated.

1. CS
2. MAs
3. Clients

Hi Mike,

OK, I drilled down the virtual client “vCenter-ccloud” to its Instance, which is called “VMware”, then I right-clicked it to bring up its properties, under its Proxies tab, a group called “Infrastructure” is there (see below screenshot).

Then, I checked this “Infrastructure” group, within it, there are 3 Servers:

1. mil-ccloud-nas (it’s dead and not in use anymore)
2. mil-commvault-2 (this is the CommServe)
3. OLSUAT (it’s only half configured and not yet in use)

Those 3 Servers are also the Media Agents (see below screenshot)

Therefore, am I right to say that, in this case, the Proxy is also the CommServe which I’ll upgrade ? In other words, just upgrading the CommServe is all I need to do ?

Thanks,

You are correct!

If that dead server is not coming back, I would deconfigure it assuming nothing is actually connected to it (as far as Commvault is concerned).

Hi Mike,

Just to let you know that the CommServe has been upgraded to 11.16.93. However, the Live Browse function still doesn’t work, so it’s up to the case engineer to escalate it to their DEV team now.

Thanks for your great help!

Appreciate you sharing!  Keep us posted (I’ll try and follow the incident closely as well!).

Hi Mike,

The Commvault support has advised that there might be wrong with our current CommServe No. 2, so they have suggested that I use another Media Agent to try the LiveBrowse.

OK, I see the point. 

But, since the current CommServe No. 2 is only only CommServe, as well as the Media Agent in this Commvault environment, I’ve create our VM administrator to create a new Windows VM for this purpose.

However, I wonder, after the creation of Windows VM, what are the minimum Commvault components do I need to install on it, apart from the obvious Media Agent and VSA ?

Thanks,

Hey @Kelvin !  looks like the case was updated, though I’m sharing here as well:

When installing, please choose the File System, Media Agent, and Virtual Server Agent packages

It sounds like some of the binaries disappeared/corrupted (though if that’s wrong, correct me).

Since it’s all the same Base code, different functions will refer to executables, etc. within other packages (that are installed if needed).  If what I suspect is true, then one of the binaries needed was missing.

That makes sense!

However, the disappearance of this File System on the CommServe is posing a dilemma here because

1. With the new MA configured for LiveBrowse, and a total backup infrastructure refresh on its way in a couple of months, it doesn’t seem very worthy to get this problem fixed...
2. On the other hand, I’m not too sure what serious problems it may cause if I leave it unfixed ?

@Kelvin , in my experience, not fixing this now will almost certainly become an issue at 2am on a Saturday night when you least expect it 

The engineer you are working with is TOP NOTCH so I’d definitely recommend working with him and getting it addressed 100%!

Thanks for the advice, Mike.

Unfortunately, the current case was closed so I’ll log a new case for this issue, but, obviously, it will be assigned to someone else...

@Kelvin reference the old case number so the new engineer (fi there is one) can see that it is related.

I’ll mark this as answered, though fi you have further questions please do share!