Skip to main content
Solved

Error 91:32 When Trying to Restore a VM


Forum|alt.badge.img+7

Hello, this might not be enough info but I’m trying to restore a VM and keep getting this error

Error Code: [91:32]
Description: Unable to register virtual machine [SERVER]. Possible reason includes datastore was not mounted correctly.
Source: CVMA01, Process: CVD

Later these errors only occur: 

Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

I have the job log files but can’t really make sense of it. Its pretty much nothing but errors and fails. 

Best answer by tsmtmiller

Mike Struening wrote:

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

 

VMware came back with an MTU size issue causing the datastore snapshots to mount but not unmount. This was breaking the restores. The VMkernel Adapter for Backups had the MTU size set to 9000. The Virtual Switch for Backups had a MTU setting of 1500. We changed the MTU’s to match. Once we did this, I was able to manually mount and unmount the snapshots and finally actually perform a full restore of the VM.

View original
Did this answer your question?

31 replies

Forum|alt.badge.img+3
  • Vaulter
  • 15 replies
  • March 29, 2021

Hello,

This looks like restore from snap operation. We will need logs to look into further but my guess some left over operation is causing this issue.

can you please check CVD.log for errors.


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • March 29, 2021

Yes, its an intellisnap backup. I see the backup get mounted into vcenter then nothing happens. I also see:

“Unable to communicate with MediaAgent on host [CVMA01]. Please check the network connectivity between the Client and MediaAgent, and verify this product's Communication Service is running on the MediaAgent.”


Forum|alt.badge.img+15

Hi @tsmtmiller 

The “Unable to communicate...” error may be a physical issue with a network firewall blocking traffic on Commvault communications ports, usually 8400-8403, or may be an issue with Network Routes configured on the client(s) or client groups.

Please check if the restore target client is able to reach the MA [CVMA01] on communications ports, you can use  cvping to test connections using specific ports:
 

cvping 191.168.1.100 -Port 8403

Full usage details for CvPing (Windows):

USAGE: <Addr> [Family] [-BindIP <IP>] [-Port #] [-n #] [-t] [-i #]
 Addr   - Interface name or IP address
 Family - Family type. Values are -UseIPv4, -UseIPv6 or -UseIPAny. Default is -UseIPv4
 -BindIP - The ip address to bind to
 -Port  - Optional port number to connect to. Default is 8400
 -n     - Number of runs
 -t - For running continuously until Ctrl+C is encountered.
 -i - Interval in seconds between two consecutive runs. Default is 1 second

 Example: cvping Machine.Comp.Com -UseIPv4 -Port 8402 -n 5 -i 2


Thanks,

Stuart


Forum|alt.badge.img+2

About this error:

“Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

This usually happens when you are trying to restore a volume or virtual machine that is currently executing another operation usually performing backup operation. you should stop any  running operation.   

 


Damian Andre
Vaulter
Forum|alt.badge.img+23

It would be good to know if this was an FC, iSCSI or NFS datastore - my guess is FC - and if so, if you have multiple adapters or WWNs, commvault could be trying to associate the snap with the wrong adapter. You can override it using this setting: https://documentation.commvault.com/additionalsetting/details?name=sPortInfo

 

That is a common scenario, but just a guess for now ;)


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • March 30, 2021

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 


Damian Andre
Vaulter
Forum|alt.badge.img+23
tsmtmiller wrote:

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?


MichaelCapon
Vaulter
Forum|alt.badge.img+14

Guessing this is VMware, I would suggest to try manually mapping the Snap to the ESX (From Array Management) and check if works okay first.

Also, If using NFS it may also be worth checking in VMware the the Host you are mapping to hasn’t reached the limit of volumes. - https://kb.vmware.com/s/article/2239

Best Regards,

Michael


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • April 5, 2021
Damian Andre wrote:
tsmtmiller wrote:

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 


Damian Andre
Vaulter
Forum|alt.badge.img+23
tsmtmiller wrote:
Damian Andre wrote:
tsmtmiller wrote:

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Got it. Inaccessible is probably coming from the job failing and then the share is removed before the datastore cleanup is performed. I do not think VMware will allow the creation of the NFS datastore initially if it's inaccessible but not sure. One thing you could try is manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host to ensure that is all working correctly ‘under the hood’


Mike Struening
Vaulter
Forum|alt.badge.img+23

@tsmtmiller , following up to see if @Damian Andre ‘s suggestion of manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host worked (or illuminated issues).

Thanks!!


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • April 7, 2021

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.


Damian Andre
Vaulter
Forum|alt.badge.img+23
tsmtmiller wrote:

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Usually from OnCommand Unified Manager - the NetApp tool that allows you to manage snapshots and all things NetApp.


Mike Struening
Vaulter
Forum|alt.badge.img+23

@tsmtmiller , gentle follow up to see how this was looking.  @Damian Andre mentioned you can use the NetApp tool OnCommand Unified Manager to manage your snapshots.  You should be able to put the snapshot online and see if ESXi can access it.


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • April 14, 2021

The system is not connected to any OnCommand Unified Manager. I had to reach out to 3rd party support who is now reaching out to CV support. 


Mike Struening
Vaulter
Forum|alt.badge.img+23

@tsmtmiller appreciate the update!  When you have the incident number, can you update the thread (or pm it to me directly)?

Thanks!!


Mike Struening
Vaulter
Forum|alt.badge.img+23

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • April 26, 2021
Mike Struening wrote:

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

 

Hello Mike, the case # is through a third party. CSPRO0096433. During a call, we saw errors in vcenter and logs showing a failure to unmount snaps from the ESX host. I asked the VMware admin to reboot all the ESX hosts. After that was done, I tried again to recover a server and once again I get the error Unable to register virtual machine. (I sent the job logs to support.) I looked again in vcenter and see commvault was able to remove the datastore of the failed restore job. So we are still looking at this. 


Mike Struening
Vaulter
Forum|alt.badge.img+23

Appreciate it.  I looked for the partner case number but it’s not showing in a search.  If you can find the CV support case number, that would help.  Otherwise, just keep us posted.

Thanks!


Mike Struening
Vaulter
Forum|alt.badge.img+23

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • May 5, 2021
Mike Struening wrote:

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!

Thanks

CV closed the case so the 3rd party support is reopening a case with them. I uploaded a new log yesterday, its the same error. I have a feeling it’s something on VCenter as I see some errors during the restore in vcenter under the monitor section:

“Error caused by file <esx backup>

“Datastore <esx backup> is not accessible . “No connected and accessible host with required privilege is attached to this datastore”

 


Mike Struening
Vaulter
Forum|alt.badge.img+23

Appreciate the update.  If you can get the CV case number I can track it.


Mike Struening
Vaulter
Forum|alt.badge.img+23

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!


Forum|alt.badge.img+7
  • Author
  • Byte
  • 35 replies
  • May 20, 2021
Mike Struening wrote:

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!

We think its an issue with VMWare. We have just started to gather logs to send to VMware. I will post an update soon. 


Mike Struening
Vaulter
Forum|alt.badge.img+23

Appreciate the update!


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings