Skip to main content
commvault.com commvault.com
Solved

Invalid VM Snapshot Mount

  • January 10, 2024
  • 8 replies
  • 481 views
dude
Community All Star
Forum|alt.badge.img+20
  • dudeCommunity All Star
  • Community All Star

Has anyone seeing this case where when running a backup copy, VMs are mounted, but some of them end up “invalid” this failing backup copy and hanging around vsphere. We have to manually right click and select “Remove from Inventory then run another backup which then works fine.

VMTools running latest version.

Run a full on the affected VMs and it keeps happening.

Had a ticket with CV which said that this is a VM issue, so wondering if the community has seen this before and have found a solution for it?

 

Thanks

Best answer by Damian Andre

I feel like @MichaelCapon  was on the money here.

 

The most likely issue here is that your storage is mapped to a group of ESXi servers (in the context of NetApp, lets call this an iGroup). Even though you have one ESXi server specified for the mount that is dissociated with other ESXi hosts in the subclient, it actually depends on how your storage is mapped from your array to your ESXi hosts. when we bring the snapshot online the other ESXi servers can ‘see’ the new storage and will attempt to automatically mount it. When we drop the snapshot, it becomes orphaned in the ESXi inventory as other ESXi are holding a handle on the snapshot datastore and it becomes a APD issue.

Commvault reuses existing relationships that have been configured on the storage side - so if your datastore LUNs are mapped to a group of ESXi hosts, the snapshots will be too when we bring them online to perform the backup copy. This is the crux of the issue.

The annoying fix to this issue is to have 1:1 mappings of datastore to ESXi hosts when configuring the lun mapping / access on your storage array, rather than to a group of hosts. This way when we bring a snapshot datastore online it wont be visible to the entire group. This is obviously a big infrastructure change, so alternatively, you can try the setting that Michael recommended.

https://documentation.commvault.com/2023e/expert/using_esx_server_for_snapshot_mount.html

I believe this setting will attempt to unmount the datastore from each esxi host before taking it offline to ensure nothing is holding on to it.

8 replies

MichaelCapon
Vaulter
Forum|alt.badge.img+19
  • MichaelCaponVaulter
  • Vaulter
  • January 10, 2024

Hi @dude,

Are you seeing any events in VMware for the Snap Mounted DS or VM? - Is this being mapped to all ESX during copy or a singular ESX?

If the Snap is mounted to a Host Group, you might need to add the nClusterMount setting (value 1) to the VSA proxy/MA to ensure its properly unmounted from all hosts,

I’d also suggest checking if the backup copy is encountering any failures/re-attempts of the Backup Phase here. - Whilst we should be doing the clean-up of the VM and removing the snap, there could be some exception here.

I’d also suggest ensuring that any backups of the vCenter VM (and any VMware / Commvault Infra) is taken outside of the backup window of VM backups. Sometimes snapshots of virtual vCenter can cause issues and Snaps of VSA Proxies can cause cleanup issues.

 

Best Regards,

Michael

    dude
    Community All Star
    Forum|alt.badge.img+20
    • dudeCommunity All Star
    • Author
    • Community All Star
    • January 10, 2024

    Thanks @MichaelCapon for the response.

    I`m using a specific ESXi Host to mount the snapshot and that is configured at the subclient level under advanced. To make sure I understand, are you proposing to NOT specify specific host and instead just leave the VCenter then add the additional setting to my VSA Group in CV?

    This is the error I see when the backup copy fails. This also does not happen to all VMs, its usually a handful while all the other ones are backed up fine.

    Error Code: [91:4]
    Description: Virtual machine [VM_GX_BACKUP] was not found. Please verify that the virtual machine still exists and that the host is in the connected state. If the virtual machine does not exist remove it from the subclient. If the virtual machine does exist reselect the virtual machine and re-add it to the subclient content.
    Source: commserve, Process: JobManager

    dude
    Community All Star
    Forum|alt.badge.img+20
    • dudeCommunity All Star
    • Author
    • Community All Star
    • April 18, 2024

    @MichaelCapon hey Mike any idea if this has been seeing internally? I upgraded to 11.32.49 and some snaps still show as invalid preventing jobs from finishing.

    Ralph
    Vaulter
    Forum|alt.badge.img+9
    • RalphVaulter
    • Vaulter
    • April 25, 2024

    Hi @dude,

    are you running pure VMware or is it on top of Nutanix?

    dude
    Community All Star
    Forum|alt.badge.img+20
    • dudeCommunity All Star
    • Author
    • Community All Star
    • August 27, 2024

    Hi @Ralph this is pure VMware. Any insight on this?

    Damian Andre
    Vaulter
    Forum|alt.badge.img+27
    • Damian AndreVaulter
    • Vaulter
    • Answer
    • August 27, 2024

    I feel like @MichaelCapon  was on the money here.

     

    The most likely issue here is that your storage is mapped to a group of ESXi servers (in the context of NetApp, lets call this an iGroup). Even though you have one ESXi server specified for the mount that is dissociated with other ESXi hosts in the subclient, it actually depends on how your storage is mapped from your array to your ESXi hosts. when we bring the snapshot online the other ESXi servers can ‘see’ the new storage and will attempt to automatically mount it. When we drop the snapshot, it becomes orphaned in the ESXi inventory as other ESXi are holding a handle on the snapshot datastore and it becomes a APD issue.

    Commvault reuses existing relationships that have been configured on the storage side - so if your datastore LUNs are mapped to a group of ESXi hosts, the snapshots will be too when we bring them online to perform the backup copy. This is the crux of the issue.

    The annoying fix to this issue is to have 1:1 mappings of datastore to ESXi hosts when configuring the lun mapping / access on your storage array, rather than to a group of hosts. This way when we bring a snapshot datastore online it wont be visible to the entire group. This is obviously a big infrastructure change, so alternatively, you can try the setting that Michael recommended.

    https://documentation.commvault.com/2023e/expert/using_esx_server_for_snapshot_mount.html

    I believe this setting will attempt to unmount the datastore from each esxi host before taking it offline to ensure nothing is holding on to it.

      Damian Andre
      Vaulter
      Forum|alt.badge.img+27

      I will also add that an alternative solution is to use proxyless ESXi mounts. This allows us to read data off snapshot datastores without putting the datastore back into the vSphere inventory.

      Its not applicable to all scenarios, but if it meets the requirements in your environment its a way more efficient and graceful way to do backup copies for VMware data:

      https://documentation.commvault.com/2023e/expert/intellisnap_backup_operations_and_backup_copy_operations_run_without_proxy_esxi_host.html

       

        dude
        Community All Star
        Forum|alt.badge.img+20
        • dudeCommunity All Star
        • Author
        • Community All Star
        • August 28, 2024

        Thanks for the suggestion @Damian Andre I`ll see how much work it will be to remap the LUNs to the ESXi hosts or perhaps map the Media Agents instead,

         

        Readiverse Academy Certifications

        description of image
        Terms & ConditionsAccessibility statement