After upgrading our ESX Proxy from 6.7 to 7.0.3c Backup Copies are not running

Hi @Scott Stockton , thanks for the post!

Quick question: is there a job that actually fails (and if so, an error message), or does it just not run?

This may be worth a support case, though any context you can provide will help 🤓

They mostly don’t run and are stuck waiting. 

I have error messages but they are attributed to changes we made to try and fix the problem. Jobs are moving again on a host that still has 6.7 installed. Its worth noting we tried multiple 7.0.3 hosts and had intermittent results. 

After the jobs clear I will try to run under a 7 host again to generate alerts.

Sounds good.  Keep us posted.

We reverted our dedicated proxy to esxi 6.7 and it worked mostly all weekend, woke up to find backup copies waiting. 1 job that failed gave us this error.

14356 2c34  07/11 07:05:13 ######## LookupExtentsInfoWithDSHost() - Extent info found with host [dc2-p-cvprxy-01.polsinelli.law] for lun [naa.624a9370fabb58501e041f4b00034930]

14356 19a8  07/11 07:05:34 ######## MountVMFSVolume() - Failed to get datastore moref for datastore:[DC2-VMPROD2-_GX_BACKUP_19494757_2_29500228]

14356 19a8  07/11 07:05:34 ######## MountVMFSVolume() - Moving with next datastore

14356 884   07/11 07:05:34 ######## LookupExtentsInfoWithDSHost() - Extent info found with host [dc2-p-cvprxy-01.polsinelli.law] for lun [naa.624a9370fabb58501e041f4b00034930]

14356 2514  07/11 07:05:34 ######## LookupExtentsInfoWithDSHost() - Extent info found with host [dc2-p-cvprxy-01.polsinelli.law] for lun [naa.624a9370fabb58501e041f4b00034930]

14356 3d0c  07/11 07:05:34 ######## MountSignaturedVolumes() - Host VMFS volume count is zero.Please check whether snap got exposed correctly to ESX server

14356 3d0c  07/11 07:05:34 ######## MountSnap() - Inside MountSnap

14356 3d0c  07/11 07:05:34 ######## MountSnap() - lun:0200fe0000624a9370fabb58501e041f4b001f11b3466c61736841

14356 3d0c  07/11 07:05:34 ######## getDatastoreMorefUsingSerialNo() -

Searching For LUN serial Number:0200fe0000624a9370fabb58501e041f4b001f11b3466c61736841

14356 1790  07/11 07:05:35 ######## getDatastoreMorefUsingSerialNo() - Datastore name:[DC2-XENPROD-SSD04]

14356 1790  07/11 07:05:35 ######## getDatastoreMorefUsingSerialNo() - Exception formatting message - Index (zero based) must be greater than or equal to zero and less than the size of the argument list. "

Datastore Name Found :{0} DC2-XENPROD-SSD04"

14356 2fd4  07/11 07:05:35 ######## MountVMFSVolume() - Failed to get datastore moref for datastore:[DC2-VMPROD2-_GX_BACKUP_19494774_2_29500108]

14356 4d40  07/11 07:05:35 ######## MountVMFSVolume() - Failed to get datastore moref for datastore:[DC2-VMPROD2-_GX_BACKUP_19494764_2_29500174]

14356 19a8  07/11 07:05:35 ######## MountVMFSVolume() - Failed to get datastore moref for datastore:[DC2-VMPROD2-_GX_BACKUP_19494757_2_29500229]

14356 884   07/11 07:05:35 ######## getDatastoreMorefUsingSerialNo() - Datastore name:[DC2-VMPROD-SSD100]

I found something for a similar case:

Checked the permissions on the service account in use and confirmed that it was missing the rename datastore permissions.

Can you try that and let me know?

Is this a permission across the whole VCenter or just isolated to the esx proxy, either way we are looking into.

The solution in the case didn’t specify, though I would check both.

Ok after watching and babysitting for several days, Every job fails looking for the same snap datastore.

OS mount failed : [VMWare Mount snapshot failed [0xEC02ECC3:{VMwareSnapOSUtil::MountSnap(2052)/MM.60611-Datastore with lun uuid:[0200fa0000624a9370fabb58501e041f4b001f4452466c61736841] not found}] (MM.60611)]

Then errors out with: Unknown VMInfo error [0].

This particular snap is consistent accross all of the failed backup copies?

I found a snap with that UUID and tried to mount it to our proxy to see if it would resolve but It errored out and wouldn’t let me mount. Again we have a case open but I am trying to get extra eyes on it as the more the merrier to get it resolved. 

Understood.  We have some sharp minds here!

Can you share the case number so I can follow it?