commvault.com commvault.com
Question

Commvault is trying to consolidate the same VM several times

  • 14 June 2023
  • 25 replies
  • 1169 views
PedroRocha
Rank
Userlevel 2
Badge +11

Hello

Apparently, after successfully backing up some VMs (VMware), commvault is requesting disk consolidation several times in a row (without waiting the first request to finish).

This sometimes is causing the VM to be unresponsive. Commvault is being proactive but it is causing downtimes to the environment.

1 - can I disable this feature (commvault starting consolidation tasks for the VMs that needs consolidation)?

2 - can Commvault skip this VM? I read that commvault would skip VMs that needs disk consolidation. But I am not sure if this is really happening.

regards,

Pedro

Pedro

25 replies

S
Rank
Userlevel 6
Badge +14

Hi @PedroRocha ,

We cannot skip this otherwise the next backup would fail.

https://kb.commvault.com/article/62743

Can you add this additional setting and see if that helps:

https://documentation.commvault.com/additionalsetting/details?name=%22nMaxWaitForTaskAttempts%22&id=2158

Best Regards,

Sebastien Merluzzi

PedroRocha
Rank
Userlevel 2
Badge +11

Hi @PedroRocha ,

We can skip this otherwise the next backup would fail.

https://kb.commvault.com/article/62743

Can you add this additional setting and see if that helps:

https://documentation.commvault.com/additionalsetting/details?name=%22nMaxWaitForTaskAttempts%22&id=2158

Best Regards,

Sebastien Merluzzi

Thanks!

 

So, that additional settings default is 3, which value to use?

I found another named bConsolidateVM and set it to false to stop this (environment is suffering a lot).

This whole story is very awkward. Commvault it causing a lot of downtimes in the environment due to this consolidation tasks in a row. It seems to be doing what it shouldn't.

I have an opened case 230614-623

take a look at this:

 

Pedro
S
Rank
Userlevel 6
Badge +14

HI @PedroRocha ,

Well Commvault is only sending the API request, same for taking/removing Snapshots, so the real question is why this is taking so long for VMWare to consolidate the VM.

You can set it to 6 for now. However I would recommend you log a case with VMware so they can investigate. Of course you can log a case with us so we can work with VMware.

Best Regards,

Sebastien Merluzzi

PedroRocha
Rank
Userlevel 2
Badge +11

cvlt dev is reviewing, I'll update it here

Pedro
S
Rank
Userlevel 6
Badge +14

So you logged a case with Vmware also?

PedroRocha
Rank
Userlevel 2
Badge +11

So you logged a case with Vmware also?

No yet. Consolidation is taking long because of factors like storage array performance and old snapshots (big deltas).

The big question is why commvault keeps sending consolidation tasks even when there is an active consolidation task in place. CVLT suspects VMware is not telling commvault to stop. I believe cvlt dev will verify and if needed we will involve VMware support.

Pedro
S
Rank
Userlevel 6
Badge +14

Correct, a few things can cause this.

Same for taking/removing snapshots, if time is over then we try again.

Let’s see what Dev say then and if we have a way to stop it.

S
Rank
Badge +2

We are also facing same issues with multiple disk consolidation jobs being triggered by service account used by commvault, however, backup has been disabled ever since VM started report disk consolidation errors.

Please let me know if there is any solution or workaround.

PedroRocha
Rank
Userlevel 2
Badge +11

Hi, 

Dev is still working on it…

I set the bConsolidateVM additional setting to false so Commvault wouldn't start consolidation tasks at all.

VMware admin is doing it...

Pedro
C
Rank
Badge

Hello,

We have the same issue in our environment. Also we have a ticket open.

Do you have NVMe Datastores presented on your VMWare ?

S
Rank
Badge +2

Hi, 

Dev is still working on it…

I set the bConsolidateVM additional setting to false so Commvault wouldn't start consolidation tasks at all.

VMware admin is doing it...

Thank you, we just added same additional settings.

C
Rank
Badge

Ensure that automount is disabled on the VSA.
 

Diskpart
automount
Automount disable
automount scrub

 

Onno van den Berg
Rank
Userlevel 7
Badge +19

@PedroRocha what was the outcome of the ticket? we occasionally have the same issue and when we consolidate the ones who failed ourselves than it is done in a split second. Upon consolidation failure it seems it is indeed really impacting the availability of the VM.

S
Rank
Userlevel 6
Badge +14

@Onno van den Berg ,

Development has created a Hotfix.

MR87 has been released which includes the hotfix that was part of MR85.

https://documentation.commvault.com/2023e/expert/assets/service_pack/updates/11_28_87.htm

Multiple disk consolidation tasks may be triggered per VM if it needs consolidation as part of different jobs

 5368
Onno van den Berg
Rank
Userlevel 7
Badge +19

I assume this fix was also created for FR30 and FR32. Can you share which MR contains the enhancement? 

S
Rank
Userlevel 6
Badge +14

It’s in our documentation in FR30/32 sections.

Onno van den Berg
Rank
Userlevel 7
Badge +19

Can you be a little bit more precise? The documentation is a rather brought big thing. No idea where to look…..

clariion
Rank
Badge +2

Hey folks,

we have the same issue with multiple customers after we did the SP Upgrade 11.28.68 to 11.28.89.

Is there any official solution?

Don't sell #GME | hold
S
Rank
Userlevel 6
Badge +14

@Onno van den Berg

Our Documentation will be updated tomorrow once the latest MR has been released:

In our Documentation go to What's New and select your Commvault Platform Release

Then go to Maintenance Releases > List of Maintenance Releases for Commvault Platform Release

You then need to search for:

Multiple disk consolidation tasks may be triggered per VM if it needs consolidation as part of different jobs

It is in 28.85, 30.72 and 32.23.

@clariion

The issue might be different as this is already fixed in 28.85.

Can you log a case so we can check and confirm?

clariion
Rank
Badge +2

 

@clariion

The issue might be different as this is already fixed in 28.85.

Can you log a case so we can check and confirm?

 

231106-382

Don't sell #GME | hold
S
Rank
Userlevel 6
Badge +14

@clariion  Perfect, will ask colleague to provide hotfix to confirm this is installed as part as MR89.

Meanwhile, you can set bConsolidateVM additional setting.

@PedroRocha  Did installing MR85+ fix it?

PedroRocha
Rank
Userlevel 2
Badge +11

@clariion  Perfect, will ask colleague to provide hotfix to confirm this is installed as part as MR89.

Meanwhile, you can set bConsolidateVM additional setting.

@PedroRocha  Did installing MR85+ fix it?

Hello, haven't tested yet

Pedro
S
Rank
Userlevel 6
Badge +14

@PedroRocha ok, or you can install 28.91 as @clariion confirmed After installing 11.28.91 all consolidation errors stopped.

Onno van den Berg
Rank
Userlevel 7
Badge +19

@Sebastien Merluzzi unfortunately there is no way to relate forward port fixes in the maintenance releases of newer versions hence this fix is not mentioned in the release notes of the maintenance releases for FR30 and FR32. Can you please share the related fix numbers so we can pinpoint which maintenance releases should contain the fix. You mentioned the maintenance releases, however we ran into issues even after being on mentioned maintenance release, so or we hit another issue related to the same which was addressed by another fix. 

S
Rank
Badge

MR: 11.32.28 / 11.32.35 which has a hotfix (8726)
https://documentation.commvault.com/v11/expert/files/service_pack/updates/11_32_28.htm

Reply


Terms & ConditionsCookie settings