commvault.com commvault.com
Solved

Error 91:32 When Trying to Restore a VM

T
Rank
Userlevel 3
Badge +6

Hello, this might not be enough info but I’m trying to restore a VM and keep getting this error

Error Code: [91:32]
Description: Unable to register virtual machine [SERVER]. Possible reason includes datastore was not mounted correctly.
Source: CVMA01, Process: CVD

Later these errors only occur: 

Error Code: [32:392]
Description: Volume Operation in Progress : [One or more volumes are being processed by other operation. VolumeId [543702]]
Source: CVMA01, Process: CVD

I have the job log files but can’t really make sense of it. Its pretty much nothing but errors and fails. 

icon

Best answer by tsmtmiller 9 June 2021, 20:02

View original

31 replies

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Very nice and detailed!  Thank you @tsmtmiller  for sharing for all posterity!

 

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

 

VMware came back with an MTU size issue causing the datastore snapshots to mount but not unmount. This was breaking the restores. The VMkernel Adapter for Backups had the MTU size set to 9000. The Virtual Switch for Backups had a MTU setting of 1500. We changed the MTU’s to match. Once we did this, I was able to manually mount and unmount the snapshots and finally actually perform a full restore of the VM.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Sounds good (and hopeful)!

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

 

Hello, I am finalizing a test to see if the solution worked. 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

Thanks in advance!!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller , hope all is well!

Any update from VMWare?

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate the update!

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!

We think its an issue with VMWare. We have just started to gather logs to send to VMware. I will post an update soon. 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller , following up to see if this issue is resolved (and how).

Thanks!!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate the update.  If you can get the CV case number I can track it.

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!

Thanks

CV closed the case so the 3rd party support is reopening a case with them. I uploaded a new log yesterday, its the same error. I have a feeling it’s something on VCenter as I see some errors during the restore in vcenter under the monitor section:

“Error caused by file <esx backup>

“Datastore <esx backup> is not accessible . “No connected and accessible host with required privilege is attached to this datastore”

 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hey @tsmtmiller .  Wasn’t able to find the CV case number.  Were you able to get this resolved?  IF so, can you share the solution here?

Thanks!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate it.  I looked for the partner case number but it’s not showing in a search.  If you can find the CV support case number, that would help.  Otherwise, just keep us posted.

Thanks!

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

 

Hello Mike, the case # is through a third party. CSPRO0096433. During a call, we saw errors in vcenter and logs showing a failure to unmount snaps from the ESX host. I asked the VMware admin to reboot all the ESX hosts. After that was done, I tried again to recover a server and once again I get the error Unable to register virtual machine. (I sent the job logs to support.) I looked again in vcenter and see commvault was able to remove the datastore of the failed restore job. So we are still looking at this. 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller , let me know if you have a case number for this issue.  I’d like to get the solution documented :sunglasses:

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller appreciate the update!  When you have the incident number, can you update the thread (or pm it to me directly)?

Thanks!!

https://www.linkedin.com/in/michael-struening
T
Rank
Userlevel 3
Badge +6

The system is not connected to any OnCommand Unified Manager. I had to reach out to 3rd party support who is now reaching out to CV support. 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller , gentle follow up to see how this was looking.  @Damian Andre mentioned you can use the NetApp tool OnCommand Unified Manager to manage your snapshots.  You should be able to put the snapshot online and see if ESXi can access it.

https://www.linkedin.com/in/michael-struening
Damian Andre
Rank
Userlevel 7
Badge +23

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Usually from OnCommand Unified Manager - the NetApp tool that allows you to manage snapshots and all things NetApp.

T
Rank
Userlevel 3
Badge +6

I have never tried manually putting the snap online from the vault copy using the NetApp tools. Where can I find the tool? I tried to ping the secondary netapp from the ESX servers and the ping appears to be working ok.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@tsmtmiller , following up to see if @Damian Andre ‘s suggestion of manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host worked (or illuminated issues).

Thanks!!

https://www.linkedin.com/in/michael-struening
Damian Andre
Rank
Userlevel 7
Badge +23

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

Got it. Inaccessible is probably coming from the job failing and then the share is removed before the datastore cleanup is performed. I do not think VMware will allow the creation of the NFS datastore initially if it's inaccessible but not sure. One thing you could try is manually putting the snap online from the vault copy using the NetApp tools and then connecting it to the ESXi host to ensure that is all working correctly ‘under the hood’

T
Rank
Userlevel 3
Badge +6

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

 I am trying to recover it to the same vcenter. Yes, I see a backup show up in vcenter “NFS _GX_BACKUP…...” but I see “inaccessible” attached to it. I cant see or read anything when clicking on it. Again it works perfectly if the snapshot resides on the vcenter but once its copied to the vault it wont restore. 

MichaelCapon
Rank
Userlevel 6
Badge +14

Guessing this is VMware, I would suggest to try manually mapping the Snap to the ESX (From Array Management) and check if works okay first.

Also, If using NFS it may also be worth checking in VMware the the Host you are mapping to hasn’t reached the limit of volumes. - https://kb.vmware.com/s/article/2239

Best Regards,

Michael

Damian Andre
Rank
Userlevel 7
Badge +23

Its NFS. I can recover from our Production system, but the vault copies on our Storage system are failing. 

The plot thickens :)

Are you recovering from the vault copy to a different vCenter instance or back to the same? Do you say you see the NFS _GX_BACKUP datastore mounted to the host you are expecting it to be mounted to?

Reply


Terms & ConditionsCookie settings