Invalid VM Snapshot Mount

Hi @dude,

Are you seeing any events in VMware for the Snap Mounted DS or VM? - Is this being mapped to all ESX during copy or a singular ESX?

If the Snap is mounted to a Host Group, you might need to add the nClusterMount setting (value 1) to the VSA proxy/MA to ensure its properly unmounted from all hosts,

I’d also suggest checking if the backup copy is encountering any failures/re-attempts of the Backup Phase here. - Whilst we should be doing the clean-up of the VM and removing the snap, there could be some exception here.

I’d also suggest ensuring that any backups of the vCenter VM (and any VMware / Commvault Infra) is taken outside of the backup window of VM backups. Sometimes snapshots of virtual vCenter can cause issues and Snaps of VSA Proxies can cause cleanup issues.

Best Regards,

Michael

Thanks @MichaelCapon for the response.

I`m using a specific ESXi Host to mount the snapshot and that is configured at the subclient level under advanced. To make sure I understand, are you proposing to NOT specify specific host and instead just leave the VCenter then add the additional setting to my VSA Group in CV?

This is the error I see when the backup copy fails. This also does not happen to all VMs, its usually a handful while all the other ones are backed up fine.

Error Code: [91:4]
Description: Virtual machine [VM_GX_BACKUP] was not found. Please verify that the virtual machine still exists and that the host is in the connected state. If the virtual machine does not exist remove it from the subclient. If the virtual machine does exist reselect the virtual machine and re-add it to the subclient content.
Source: commserve, Process: JobManager

@MichaelCapon hey Mike any idea if this has been seeing internally? I upgraded to 11.32.49 and some snaps still show as invalid preventing jobs from finishing.

Hi @dude,

are you running pure VMware or is it on top of Nutanix?

Hi @Ralph this is pure VMware. Any insight on this?

I feel like @MichaelCapon  was on the money here.

The most likely issue here is that your storage is mapped to a group of ESXi servers (in the context of NetApp, lets call this an iGroup). Even though you have one ESXi server specified for the mount that is dissociated with other ESXi hosts in the subclient, it actually depends on how your storage is mapped from your array to your ESXi hosts. when we bring the snapshot online the other ESXi servers can ‘see’ the new storage and will attempt to automatically mount it. When we drop the snapshot, it becomes orphaned in the ESXi inventory as other ESXi are holding a handle on the snapshot datastore and it becomes a APD issue.

Commvault reuses existing relationships that have been configured on the storage side - so if your datastore LUNs are mapped to a group of ESXi hosts, the snapshots will be too when we bring them online to perform the backup copy. This is the crux of the issue.

The annoying fix to this issue is to have 1:1 mappings of datastore to ESXi hosts when configuring the lun mapping / access on your storage array, rather than to a group of hosts. This way when we bring a snapshot datastore online it wont be visible to the entire group. This is obviously a big infrastructure change, so alternatively, you can try the setting that Michael recommended.

https://documentation.commvault.com/2023e/expert/using_esx_server_for_snapshot_mount.html

I believe this setting will attempt to unmount the datastore from each esxi host before taking it offline to ensure nothing is holding on to it.

I will also add that an alternative solution is to use proxyless ESXi mounts. This allows us to read data off snapshot datastores without putting the datastore back into the vSphere inventory.

Its not applicable to all scenarios, but if it meets the requirements in your environment its a way more efficient and graceful way to do backup copies for VMware data:

https://documentation.commvault.com/2023e/expert/intellisnap_backup_operations_and_backup_copy_operations_run_without_proxy_esxi_host.html

Thanks for the suggestion @Damian Andre I`ll see how much work it will be to remap the LUNs to the ESXi hosts or perhaps map the Media Agents instead,