Timeout waiting for HotAdd Mutex, VDDK Deadlock encountered

Hello ​@Aleksandra 

Check the vmx file of the access node (you can download it from vCenter) and confirm there are 4 SCSI Paravirtual adaptors present, and the order is as follows: 

scsi0.pciSlotNumber = "160"

scsi1.pciSlotNumber = "1184"

scsi2.pciSlotNumber = "224"

scsi3.pciSlotNumber = "256"

Additionally, check if automount is disable and scrubbed on the access node. 

Open cmd with admin on access node > diskpart > automount, if it shows enabled, run: automount scrub and automount disable. 

Then follow this: 

1. Make sure there are no jobs running for the VSA. 2. Stop the CommVault Services on the Proxy 3. Navigate to C:\Program Files\CommVault\Simpana\Base\Temp\vmware-SYSTEM\ - Clear all files/folders within "vmware-system" 

Make sure the sequence is fine, and you have cleaned up the files and if you still face the issue, share the vsbkp.log from the access node along with the job id for review. If you do not want to share it here, please feel free to raise a case.

Best,

Rajiv Singal

Hi Rajiv, 

thanks for the update.

We will try to check this, however, this will take a certain time to check on the VMWare side.

Best regards, 

Aleksandra

Hello Rajiv, 

the customer tried to implement all the recommendations you gave. 

At the step of disabling Automount, he faced the problem, that the Proxies did not boot anymore. 

From the config file they have this warning that proved to be true

 Configuration option global/event_activation.

 Disable event based autoactivation commands.

 WARNING: setting this to zero may cause machine startup to fail.

 Previously, setting this to zero would enable static autoactivation

 services (via the lvm2-activation-generator), but the autoactivation

 services and generator have been removed.

 This configuration option has an automatic default value.

They are running Commvault version 11.28.109.
And the VSA Proxies are Oracle Linux 8 and 9. 
So, the customer followed the OS documentation how to disable Automount - this is said so in the Commvault Documentation:
Disabling Automatic Activation of Volumes for a Linux Proxy
 

Best regards, 

Aleksandra

Hello ​@Aleksandra I think the proxies failed to boot due to the SCSI order being changed (if it was done) and not due to the automount. This should be checked with VM vendor (E.g. VMware) as how to prevent this from happening with the correct boot order mentioned above. 

Best,

Rajiv Singal

Hi ​@Rajiv,

thanks for your feedback.

I will let the customer know about that.

Best regards, 

Aleksandra

Hi ​@Rajiv ,

the customer checked the vmx files of the proxies. and there is only one paravirtual adapter present:

scsi0.pciSlotNumber = "160"

Do you have any idea?

Best regards, 

Aleksandra

​@Aleksandra is the access node booting now? If not, check if there any stale disk attached to it from the vCenter and perform the consolidation manually. 

Once the VM is up, make sure on each access node you have 4 paravirtual SCSI adaptors attached to them and the order should be as follows: 

scsi0.pciSlotNumber = "160"

scsi1.pciSlotNumber = "1184"

scsi2.pciSlotNumber = "224"

scsi3.pciSlotNumber = "256"

Increasing the adaptors will be helpful for the VDDK thread timeout issues.

Best,

Rajiv Singal

Hi ​@Rajiv,

thanks for your reply. 

Do you mean, that the  customer should manually add the missing PVSCSI adaptors into the vmx file?

Regards, 

Aleksandra

​@Aleksandra if customer is not aware of how to add more SCSI adaptors to a machine and change the order, it's better to raise a case with Broadcom for assistance or check with your VMware admin on the same. 

Best,

Rajiv Singal

Got it, thank you