Question

Error 91:32 When Trying to Restore a VM


Badge +3

Hello, this might not be enough info but I’m trying to restore a VM and keep getting this error

Error Code: [91:32]
Description: Unable to register virtual machine [SERVER]. Possible reason includes datastore was not mounted correctly.
Source: CVMA01, Process: CVD

Later these errors only occur: 

Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

I have the job log files but can’t really make sense of it. Its pretty much nothing but errors and fails. 


16 replies

Userlevel 6
Badge +14

@tsmtmiller appreciate the update!  When you have the incident number, can you update the thread (or pm it to me directly)?

Thanks!!

Badge +3

The system is not connected to any OnCommand Unified Manager. I had to reach out to 3rd party support who is now reaching out to CV support. 

Userlevel 6
Badge +14

@tsmtmiller , gentle follow up to see how this was looking.  @Damian Andre mentioned you can use the NetApp tool OnCommand Unified Manager to manage your snapshots.  You should be able to put the snapshot online and see if ESXi can access it.

Userlevel 5
Badge +13

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Usually from OnCommand Unified Manager - the NetApp tool that allows you to manage snapshots and all things NetApp.

Badge +3

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Userlevel 6
Badge +14

@tsmtmiller , following up to see if @Damian Andre ‘s suggestion of manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host worked (or illuminated issues).

Thanks!!

Userlevel 5
Badge +13

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Got it. Inaccessible is probably coming from the job failing and then the share is removed before the datastore cleanup is performed. I do not think VMware will allow the creation of the NFS datastore initially if it's inaccessible but not sure. One thing you could try is manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host to ensure that is all working correctly ‘under the hood’

Badge +3

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Userlevel 4
Badge +7

Guessing this is VMware, I would suggest to try manually mapping the Snap to the ESX (From Array Management) and check if works okay first.

Also, If using NFS it may also be worth checking in VMware the the Host you are mapping to hasn’t reached the limit of volumes. - https://kb.vmware.com/s/article/2239

Best Regards,

Michael

Userlevel 5
Badge +13

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

Badge +3

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

Userlevel 5
Badge +13

It would be good to know if this was an FC, iSCSI or NFS datastore - my guess is FC - and if so, if you have multiple adapters or WWNs, commvault could be trying to associate the snap with the wrong adapter. You can override it using this setting: https://documentation.commvault.com/additionalsetting/details?name=sPortInfo

 

That is a common scenario, but just a guess for now ;)

Badge +1

About this error:

“Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

This usually happens when you are trying to restore a volume or virtual machine that is currently executing another operation usually performing backup operation. you should stop any  running operation.   

 

Userlevel 5
Badge +10

Hi @tsmtmiller 

The “Unable to communicate...” error may be a physical issue with a network firewall blocking traffic on Commvault communications ports, usually 8400-8403, or may be an issue with Network Routes configured on the client(s) or client groups.

Please check if the restore target client is able to reach the MA [CVMA01] on communications ports, you can use  cvping to test connections using specific ports:
 

cvping 191.168.1.100 -Port 8403

Full usage details for CvPing (Windows):

USAGE: <Addr> [Family] [-BindIP <IP>] [-Port #] [-n #] [-t] [-i #]
Addr - Interface name or IP address
Family - Family type. Values are -UseIPv4, -UseIPv6 or -UseIPAny. Default is -UseIPv4
-BindIP - The ip address to bind to
-Port - Optional port number to connect to. Default is 8400
-n - Number of runs
-t - For running continuously until Ctrl+C is encountered.
-i - Interval in seconds between two consecutive runs. Default is 1 second

Example: cvping Machine.Comp.Com -UseIPv4 -Port 8402 -n 5 -i 2


Thanks,

Stuart

Badge +3

Yes, its an intellisnap backup. I see the backup get mounted into vcenter then nothing happens. I also see:

“Unable to communicate with MediaAgent on host [CVMA01]. Please check the network connectivity between the Client and MediaAgent, and verify this product's Communication Service is running on the MediaAgent.”

Badge +2

Hello,

This looks like restore from snap operation. We will need logs to look into further but my guess some left over operation is causing this issue.

can you please check CVD.log for errors.

Reply