Live Browse for Windows VMs failed with error code -6
Hi,
We have 2 Windows Commvault Servers.
Previously, none of them could do a LiveBrowse for Windows VMs during a Guest Files and Folders restore task. After we put Commvault exclusion on our anti-virus (SentinelOne), and turned off Firewalls on them, the LiveBrowse is now working on Windows Commvault Server No.1, but still not working on the Windows Commvault Server No.2 which always fails with LiveBrowse error code -6, failed to initialize VM disks.
The 2 Servers are very similar except that
The No. 1 uses local disks as disk library, whereas No. 1 uses a NAS as disk library.
The No. 1 Commvault is 11.20, whereas No. 2 Commvault version is 11.14
Just to see if the NAS is the issue for No. 2, I’ve setup a local drive as disk library for a test backup, followed by a LiveBrowse restore, but failed again with the same error message.
I’ve logged a case with Commvault support, but the engineer couldn’t figure out why either so he has decided to escalate the case, however, I was told to upgrade the No. 2 to at least version 16…
Since there is an imminent backup system refresh on the way, so I’d prefer to keep things simple for now, therefore, I wonder if anyone on this forum could point me to some right direction, without me having to plan an upgrade ?
Many thanks,
Part of the error information in Browse.log writes as below
“
22648 7764 03/15 15:21:07 ### BrowseRequestHandler::processBrowseRequest() - START - browse request with worker id 55]
22648 7764 03/15 15:21:07 ### RESTORE restore: all components indices required are in the cache
22648 7764 03/15 15:21:07 ### RESTORE open directory #D:\Program Files\Commvault\ContentStore\IndexCache\CV_Index\2\30\1615400239] to check for metadata collection flag
22648 7764 03/15 15:21:07 ### BrowseVirtualServer() This is Virtual Server EXTENT browse and the backup type is DISK backup
22648 7764 03/15 15:21:07 ### VirtualServerExtentBrowse::Browse() Virtual server Extent browse - changing emFunction from e4] to n7]
22648 7764 03/15 15:21:07 ### EMHANDLER Parent node (15) was found for Subclient '', Path='\5006bb64-d4a7-2e4f-a840-178ed2b3f384\scsi0-0-ML-RAS3.vmdk'.
22648 7764 03/15 15:21:07 ### SEARCHFORCHILDREN Query 00], 101 nodes were found in Ctree. 100 nodes were qualified after filtering
22648 7764 03/15 15:21:07 ### BrowseRequestHandler::processBrowseRequest() - END - browse request with worker id -55]
22648 43a8 03/15 15:21:07 ### VMLivebrowseClient::ExecuteRemoteCall() - wVMLivebrowse] Received response back from Live browse.
22648 43a8 03/15 15:21:07 ### VMLivebrowseClient::MountVM() - #VMLivebrowse] Remote error during VMountVM] : 234881285, Failed to intialize VM disks
22648 43a8 03/15 15:21:07 ### VMLivebrowseClient::FileSystemEnumerateItems() - iVMLivebrowse] Failed to mount VM 5006bb64-d4a7-2e4f-a840-178ed2b3f384]
22648 43a8 03/15 15:21:07 ### LiveBrowseVirtualServer() Live browse enumeration Enumeration failed: 234881285 Failed to intialize VM disks
22648 43a8 03/15 15:21:07 ### RESTORE external handler failed with code -6 22648 43a8 03/15 15:21:07 ### BrowseRequestHandler::browse() - browse request failed 22648 43a8 03/15 15:21:07 ### released session mil-commvault-2.ccloud.neopost.com*mil-commvault-2*8400*8402:BrowseRouter] to pool
7120 86a0 03/15 15:21:07 ### FBCSWORKER received response from media agent
22648 43a8 03/15 15:21:07 ### BrowseRequestHandler::processBrowseRequest() - END - browse request with worker id e52]
7120 86a0 03/15 15:21:07 ### FBCSWORKER Warning: Prediction failed for worker id S52] 7120 86a0 03/15 15:21:07 ### FBCSWORKER END - browse request with worker id #52]
”
Page 1 / 1
Hi Kelvin,
I’d suggest checking CVD.log and PSEUDOMOUNT.log on mil-commvault-2 at/before 03/15 15:21:07 for any errors/messages that may be related. I would also suggest ensuring automount is disabled on the MA machine also.
Kind Regards,
Michael
Hi Michael,
Thanks for you quick reply.
I just did a quick check but still not sure where to start ?
For example,
Automount is disabled (in fact, it was enabled but then, disabled by the Commvault support)
PSEUDOMOUNT.log shows that it failed to initialize and mount the PSEUDO disk, if I read the below log correctly ?
“
15:21:07 ### CPseudoVMManager::CreateAndMountPseudoDevices(168) - Partial drive listing. Skipping disk since it failed to initialize: device=5006bb64-d4a7-2e4f-a840-178ed2b3f384_41145_scsi0-0-ML-RAS3.vmdk, error=0x80070001:{CVSAPseudoDevice::CreatePseudoDisk(123)/Disk Initialize failed} + {CVSADisk::Initialize(739)} + {CVirtualDiskBase::Initialize(136)/ReadAheadExtents failed} + {CVSADisk::ReadAheadExtents(676)} + {CVirtualDiskBase::ReadAheadExtents(476)/Recall failed!} + {CVSAPseudoDevice::Recall(217)} + {CVSAPseudoDevice::PostRecallTask(293)} + {CPseudoDevice::PostRecallTask(640)} + {CVirtualDiskBase::Recall(519)} + {CVirtualDiskBase::RecallInternal(535)/RecallExtent failed!} + {CVSADisk::RecallDataFromSource(559)} + {CVSADisk::RecallDataFromSource(554)} + {CRecaller::RecallExtent(298)} + {CRecaller::RecallExtent(255)} + {CRecaller::SubmitRecallRequest(378)/W32.1.(Incorrect function.)-Failed to submit restore request to DMRService, error=29}
Hey @Kelvin ! Thanks for the post. Is there any reason you don’t want to upgrade the other client to 11.20? 11.16 is the lowest supported version, so I can see that Customer Support gave a best effort, though they can only do so much.
I’d certainly like to see if anyone else has any experience, though the upgrade to 11.20 is not a complex process.
Hi Mike,
This is because I’m not too sure how relevant the version is in this case, unless this LiveBrowse has never worked before with 11.14 on our CommServe No.2...
That being said, this might be the case as I’ve just got this job 1 month ago and my predecessor doesn’t seem to be able to recall when was the last time this function still worked LOL
But, even though, as an educated guess, I’d assume that at the time when this CommServe No. 2 was installed, if configured properly, there shouldn’t have been any reason why this function would have not worked ?
So, I guess that I’m just trying to rewind the time back to 4 years ago, and try to find the old way to fix the problem LOL
Fair enough @Kelvin One thing I would check though is the history of this client and see if this EVER worked. I have absolutely seen plenty of setups where it never worked, it was pushed to the side, then X years later someone gives it a go and wonders why it stopped working. Try a Job summary report and see what you find.
I did find your ticket and made an internal note so I could track it, though as you mentioned it is waiting on an update to 11.16 or higher (and since the other one is on 11.20, I’d go that route).
Other incidents I found with your error were resolved by disabling AV (it was locking up binaries) or uninstalling and reinstalling.
If you can find when it worked before, that would help, though keep in mind, Support is looking to send this to development for review, and dev only makes updates for supported features (11.16 is the minimum long term and 11.20 is the higher long term release).
Keep me posted and let me know if I can help in any way!
Hi Mike,
Having tried to completely disable the SentinelOne on the CommServe, as well as to enable the NFS on the NAS that makes up the disk library, the LiveBrowse still doesn’t work with the same error message.
So, we are now going to upgrade the CommServe to SP16 first so as to get the Commvault support back in again.
Therefore, could I ask you the below questions:
1) Our vCentre Server version is at 6.0.0, which is pretty old, is it compatible with SP16 ?
2) Do you know, roughly, how long the upgrade take ? Knowing this would help me to decide whether to do the upgrade on a week day or at a weekend.
3) Is the upgrade itself very simple and straight-forward ? For example, do I need to read a manual/guide of it ?
Sorry for asking those very basic questions, as I'm from Netbackup background, and their upgrade was never as simple as what they said on the tin…
Thanks,
Sorry, Mike, I have one more silly question.
After upgrading to SP16, if the CommServe has some major backup issues, can we roll back to the current SP14 ? Or, we will be rather stuck with the new version ?
No silly questions at all! this is 100% the perfect place for this.
ROUGHLY….maybe 15-20 minutes per server? That’s a touch one to answer, though you’ll want to do the CS first, then the Media Agents (mostly likely) followed by any clients….and only some phases can be done simultaeously
You can always reinstall on 14 and restore a DB from 14 (I would definitely run a DR backup before you do the upgrade), but I also would expect a smooth process.
I do want to reiterate that 11.16 is the oldest Long Term Release, so I would suggest 11.20 if possible. If you go to 11.16, you’ll find yourself needing to upgrade before you know it, but 11.20 will future proof you for a long time!
Keep me posted!
Hi Mike,
I’ve downloaded the SP16 and its 1.6GB hotfix, and I think I’m almost ready for the upgrade (I’ll take it as my first experience before feeling confident enough to do it again, i.e. upgrading it to SP20.36).
Our system is quite simple, just one CommServe with 2 dead Media Agents (so I don’t have to worry about them), and very few Clients of no importance.
The key thing, however, is the vCentre client defined in Client Computers.
Having checked the property of this vCentre Client, unlike other Clients, I couldn’t find any option to show its current version.
Does this mean that I don’t need to do anything to this vCentre Client during the upgrade as the upgrade to the CommServe itself should take care of it ?
Thanks,
Most likely, you are looking at the virtual client, not the Proxy that accesses the vms. the Proxy should have a version.
If your environment is small, just update all clients once the CS, then MAs are updated.
CS
MAs
Clients
Hi Mike,
OK, I drilled down the virtual client “vCenter-ccloud” to its Instance, which is called “VMware”, then I right-clicked it to bring up its properties, under its Proxies tab, a group called “Infrastructure” is there (see below screenshot).
Then, I checked this “Infrastructure” group, within it, there are 3 Servers:
mil-ccloud-nas (it’s dead and not in use anymore)
mil-commvault-2 (this is the CommServe)
OLSUAT (it’s only half configured and not yet in use)
Those 3 Servers are also the Media Agents (see below screenshot)
Therefore, am I right to say that, in this case, the Proxy is also the CommServe which I’ll upgrade ? In other words, just upgrading the CommServe is all I need to do ?
Thanks,
You are correct!
If that dead server is not coming back, I would deconfigure it assuming nothing is actually connected to it (as far as Commvault is concerned).
Hi Mike,
Just to let you know that the CommServe has been upgraded to 11.16.93. However, the Live Browse function still doesn’t work, so it’s up to the case engineer to escalate it to their DEV team now.
Thanks for your great help!
Appreciate you sharing! Keep us posted (I’ll try and follow the incident closely as well!).
Hi Mike,
The Commvault support has advised that there might be wrong with our current CommServe No. 2, so they have suggested that I use another Media Agent to try the LiveBrowse.
OK, I see the point.
But, since the current CommServe No. 2 is only only CommServe, as well as the Media Agent in this Commvault environment, I’ve create our VM administrator to create a new Windows VM for this purpose.
However, I wonder, after the creation of Windows VM, what are the minimum Commvault components do I need to install on it, apart from the obvious Media Agent and VSA ?
Thanks,
Hey @Kelvin ! looks like the case was updated, though I’m sharing here as well:
When installing, please choose the File System, Media Agent, and Virtual Server Agent packages
Hi Mike,
Just to give a quick update.
After setting up a new Windows MA and using it for LiveBrowse Windows VM, the function now is working again.
The cause of the problem has also been identified, it is “the disappearance” of File System agent on the CommServe itself, although it is installed.
However, i just wonder how this “disappearance of File System” caused the failure of a LiveBrowse ?
Cheers,
It sounds like some of the binaries disappeared/corrupted (though if that’s wrong, correct me).
Since it’s all the same Base code, different functions will refer to executables, etc. within other packages (that are installed if needed). If what I suspect is true, then one of the binaries needed was missing.
That makes sense!
However, the disappearance of this File System on the CommServe is posing a dilemma here because
With the new MA configured for LiveBrowse, and a total backup infrastructure refresh on its way in a couple of months, it doesn’t seem very worthy to get this problem fixed...
On the other hand, I’m not too sure what serious problems it may cause if I leave it unfixed ?
@Kelvin , in my experience, not fixing this now will almost certainly become an issue at 2am on a Saturday night when you least expect it
The engineer you are working with is TOP NOTCH so I’d definitely recommend working with him and getting it addressed 100%!
Thanks for the advice, Mike.
Unfortunately, the current case was closed so I’ll log a new case for this issue, but, obviously, it will be assigned to someone else...
@Kelvin reference the old case number so the new engineer (fi there is one) can see that it is related.
I’ll mark this as answered, though fi you have further questions please do share!