commvault.com commvault.com
Solved

Error 91:32 When Trying to Restore a VM

T
Rank
Userlevel 3
Badge +6

Hello, this might not be enough info but I’m trying to restore a VM and keep getting this error

Error Code: [91:32]
Description: Unable to register virtual machine [SERVER]. Possible reason includes datastore was not mounted correctly.
Source: CVMA01, Process: CVD

Later these errors only occur: 

Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

I have the job log files but can’t really make sense of it. Its pretty much nothing but errors and fails. 

icon

Best answer by tsmtmiller 9 June 2021, 20:02

View original

31 replies

S
Rank
Userlevel 7
Badge +15

Hi @tsmtmiller 

The “Unable to communicate...” error may be a physical issue with a network firewall blocking traffic on Commvault communications ports, usually 8400-8403, or may be an issue with Network Routes configured on the client(s) or client groups.

Please check if the restore target client is able to reach the MA [CVMA01] on communications ports, you can use  cvping to test connections using specific ports:
 

cvping 191.168.1.100 -Port 8403

Full usage details for CvPing (Windows):

USAGE: <Addr> [Family] [-BindIP <IP>] [-Port #] [-n #] [-t] [-i #]
 Addr   - Interface name or IP address
 Family - Family type. Values are -UseIPv4, -UseIPv6 or -UseIPAny. Default is -UseIPv4
 -BindIP - The ip address to bind to
 -Port  - Optional port number to connect to. Default is 8400
 -n     - Number of runs
 -t - For running continuously until Ctrl+C is encountered.
 -i - Interval in seconds between two consecutive runs. Default is 1 second

 Example: cvping Machine.Comp.Com -UseIPv4 -Port 8402 -n 5 -i 2


Thanks,

Stuart

Customer Support
T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

 

Hello, I am finalizing a test to see if the solution worked. 

T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!

We think its an issue with VMWare. We have just started to gather logs to send to VMware. I will post an update soon. 

T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

 

VMware came back with an MTU size issue causing the datastore snapshots to mount but not unmount. This was breaking the restores. The VMkernel Adapter for Backups had the MTU size set to 9000. The Virtual Switch for Backups had a MTU setting of 1500. We changed the MTU’s to match. Once we did this, I was able to manually mount and unmount the snapshots and finally actually perform a full restore of the VM.

Damian Andre
Rank
Userlevel 7
Badge +23

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Usually from OnCommand Unified Manager - the NetApp tool that allows you to manage snapshots and all things NetApp.

A
Rank
Userlevel 2
Badge +3

Hello,

This looks like restore from snap operation. We will need logs to look into further but my guess some left over operation is causing this issue.

can you please check CVD.log for errors.

M
Rank
Badge +2

About this error:

“Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

This usually happens when you are trying to restore a volume or virtual machine that is currently executing another operation usually performing backup operation. you should stop any  running operation.   

 

Damian Andre
Rank
Userlevel 7
Badge +23

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller , gentle follow up to see how this was looking.  @Damian Andre mentioned you can use the NetApp tool OnCommand Unified Manager to manage your snapshots.  You should be able to put the snapshot online and see if ESXi can access it.

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

T
Rank
Userlevel 3
Badge +6

The system is not connected to any OnCommand Unified Manager. I had to reach out to 3rd party support who is now reaching out to CV support. 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Sounds good (and hopeful)!

https://www.linkedin.com/in/michael-struening
Damian Andre
Rank
Userlevel 7
Badge +23

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Got it. Inaccessible is probably coming from the job failing and then the share is removed before the datastore cleanup is performed. I do not think VMware will allow the creation of the NFS datastore initially if it's inaccessible but not sure. One thing you could try is manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host to ensure that is all working correctly ‘under the hood’

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate the update!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller appreciate the update!  When you have the incident number, can you update the thread (or pm it to me directly)?

Thanks!!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Very nice and detailed!  Thank you @tsmtmiller  for sharing for all posterity!

 

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

 

Hello Mike, the case # is through a third party. CSPRO0096433. During a call, we saw errors in vcenter and logs showing a failure to unmount snaps from the ESX host. I asked the VMware admin to reboot all the ESX hosts. After that was done, I tried again to recover a server and once again I get the error Unable to register virtual machine. (I sent the job logs to support.) I looked again in vcenter and see commvault was able to remove the datastore of the failed restore job. So we are still looking at this. 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller , following up to see if @Damian Andre ‘s suggestion of manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host worked (or illuminated issues).

Thanks!!

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

T
Rank
Userlevel 3
Badge +6

Yes, its an intellisnap backup. I see the backup get mounted into vcenter then nothing happens. I also see:

“Unable to communicate with MediaAgent on host [CVMA01]. Please check the network connectivity between the Client and MediaAgent, and verify this product's Communication Service is running on the MediaAgent.”

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate it.  I looked for the partner case number but it’s not showing in a search.  If you can find the CV support case number, that would help.  Otherwise, just keep us posted.

Thanks!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!

https://www.linkedin.com/in/michael-struening

Reply


Terms & ConditionsCookie settings