Solved

Error 91:32 When Trying to Restore a VM


Userlevel 3
Badge +6

Hello, this might not be enough info but I’m trying to restore a VM and keep getting this error

Error Code: [91:32]
Description: Unable to register virtual machine [SERVER]. Possible reason includes datastore was not mounted correctly.
Source: CVMA01, Process: CVD

Later these errors only occur: 

Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

I have the job log files but can’t really make sense of it. Its pretty much nothing but errors and fails. 

icon

Best answer by tsmtmiller 9 June 2021, 20:02

View original

31 replies

Userlevel 2
Badge +3

Hello,

This looks like restore from snap operation. We will need logs to look into further but my guess some left over operation is causing this issue.

can you please check CVD.log for errors.

Userlevel 3
Badge +6

Yes, its an intellisnap backup. I see the backup get mounted into vcenter then nothing happens. I also see:

“Unable to communicate with MediaAgent on host [CVMA01]. Please check the network connectivity between the Client and MediaAgent, and verify this product's Communication Service is running on the MediaAgent.”

Userlevel 7
Badge +15

Hi @tsmtmiller 

The “Unable to communicate...” error may be a physical issue with a network firewall blocking traffic on Commvault communications ports, usually 8400-8403, or may be an issue with Network Routes configured on the client(s) or client groups.

Please check if the restore target client is able to reach the MA [CVMA01] on communications ports, you can use  cvping to test connections using specific ports:
 

cvping 191.168.1.100 -Port 8403

Full usage details for CvPing (Windows):

USAGE: <Addr> [Family] [-BindIP <IP>] [-Port #] [-n #] [-t] [-i #]
Addr - Interface name or IP address
Family - Family type. Values are -UseIPv4, -UseIPv6 or -UseIPAny. Default is -UseIPv4
-BindIP - The ip address to bind to
-Port - Optional port number to connect to. Default is 8400
-n - Number of runs
-t - For running continuously until Ctrl+C is encountered.
-i - Interval in seconds between two consecutive runs. Default is 1 second

Example: cvping Machine.Comp.Com -UseIPv4 -Port 8402 -n 5 -i 2


Thanks,

Stuart

Badge +2

About this error:

“Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

This usually happens when you are trying to restore a volume or virtual machine that is currently executing another operation usually performing backup operation. you should stop any  running operation.   

 

Userlevel 7
Badge +23

It would be good to know if this was an FC, iSCSI or NFS datastore - my guess is FC - and if so, if you have multiple adapters or WWNs, commvault could be trying to associate the snap with the wrong adapter. You can override it using this setting: https://documentation.commvault.com/additionalsetting/details?name=sPortInfo

 

That is a common scenario, but just a guess for now ;)

Userlevel 3
Badge +6

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

Userlevel 7
Badge +23

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

Userlevel 6
Badge +14

Guessing this is VMware, I would suggest to try manually mapping the Snap to the ESX (From Array Management) and check if works okay first.

Also, If using NFS it may also be worth checking in VMware the the Host you are mapping to hasn’t reached the limit of volumes. - https://kb.vmware.com/s/article/2239

Best Regards,

Michael

Userlevel 3
Badge +6

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Userlevel 7
Badge +23

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Got it. Inaccessible is probably coming from the job failing and then the share is removed before the datastore cleanup is performed. I do not think VMware will allow the creation of the NFS datastore initially if it's inaccessible but not sure. One thing you could try is manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host to ensure that is all working correctly ‘under the hood’

Userlevel 7
Badge +23

@tsmtmiller , following up to see if @Damian Andre ‘s suggestion of manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host worked (or illuminated issues).

Thanks!!

Userlevel 3
Badge +6

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Userlevel 7
Badge +23

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Usually from OnCommand Unified Manager - the NetApp tool that allows you to manage snapshots and all things NetApp.

Userlevel 7
Badge +23

@tsmtmiller , gentle follow up to see how this was looking.  @Damian Andre mentioned you can use the NetApp tool OnCommand Unified Manager to manage your snapshots.  You should be able to put the snapshot online and see if ESXi can access it.

Userlevel 3
Badge +6

The system is not connected to any OnCommand Unified Manager. I had to reach out to 3rd party support who is now reaching out to CV support. 

Userlevel 7
Badge +23

@tsmtmiller appreciate the update!  When you have the incident number, can you update the thread (or pm it to me directly)?

Thanks!!

Userlevel 7
Badge +23

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

Userlevel 3
Badge +6

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

 

Hello Mike, the case # is through a third party. CSPRO0096433. During a call, we saw errors in vcenter and logs showing a failure to unmount snaps from the ESX host. I asked the VMware admin to reboot all the ESX hosts. After that was done, I tried again to recover a server and once again I get the error Unable to register virtual machine. (I sent the job logs to support.) I looked again in vcenter and see commvault was able to remove the datastore of the failed restore job. So we are still looking at this. 

Userlevel 7
Badge +23

Appreciate it.  I looked for the partner case number but it’s not showing in a search.  If you can find the CV support case number, that would help.  Otherwise, just keep us posted.

Thanks!

Userlevel 7
Badge +23

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!

Userlevel 3
Badge +6

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!

Thanks

CV closed the case so the 3rd party support is reopening a case with them. I uploaded a new log yesterday, its the same error. I have a feeling it’s something on VCenter as I see some errors during the restore in vcenter under the monitor section:

“Error caused by file <esx backup>

“Datastore <esx backup> is not accessible . “No connected and accessible host with required privilege is attached to this datastore”

 

Userlevel 7
Badge +23

Appreciate the update.  If you can get the CV case number I can track it.

Userlevel 7
Badge +23

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!

Userlevel 3
Badge +6

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!

We think its an issue with VMWare. We have just started to gather logs to send to VMware. I will post an update soon. 

Userlevel 7
Badge +23

Appreciate the update!

Reply