Skip to main content
commvault.com commvault.com

Hello!

We are finally testing Indexing v2 for VMs and it seems like it is MUCH faster than V1. We are testing on our 800+ dev/test VMs and we have some random errors:

Failure Reason: Mismatch detected in blocks being backed up for certain volumes or disks. The next incremental will resolve the situation automatically


So yeah, I opened a new case with CV support but… it seems like they have no idea what should be done :)

So, now the Support Engineer says that the problem is ...with the incremental backup before Synch Full. Interesting, but the last incremental is green. 

I am uploading more and more logs for investigation, and there is no answer syet].

So, I wanted to ask the community, probably somebody knows what could be the reason.

BTW, using Indexing v1 with the same ESXi hosts, the same network etc we did not have issues like this.

Good afternoon.  A few questions for you:

  1. Does running an incremental resolve the issue
  2. Are you seeing any errors related to disk consolidation?
  3. Has support had you run a Data Verification job and if so, did it show jobs as failing the verification?

Good afternoon.  A few questions for you:

  1. Does running an incremental resolve the issue
  2. Are you seeing any errors related to disk consolidation?
  3. Has support had you run a Data Verification job and if so, did it show jobs as failing the verification?

1 - all incremental backups are green (some with a warning that the VM could not be quiesced but nothing critical. The next Synth Full fails for random VMs. This weekend it could be 5 Vms, the next one - 10, ans they are not the same.

 

2 - nope

 

3 - we run every day incremental data verification jobs, sometimes we do have errors but they are for Files DDBs. No errors for VM DDBs as far as I remember. We could run a full scan, but we were not told to do that

Hi @Roman Melekh

From the Ticket I can see that the engineer is working with a T2 Engineer.

There is now a new issue for your Incremental Backups “VDDK thread timeout, possible VDDK deadlock encountered” and looks like you may need to involve VMWare support.

The T1 Engineer is currently reviewing the logs, he will then escalate this case to T2 and then to Development if needed.

Best Regards,

Sebastien

Now I checked the logs and I don’t think VMWare would help here.

I have seen similar issues for VDDK Deadock where VMWare would not help as the OS is not supported:

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/1ad87f77-0120-44f0-9517-4da7a9161679/3592e29c-8bd5-48f9-8e95-641b7ac0bbad/VDDK-703-ReleaseNotes.html

The following are newly supported as operating systems for HotAdd proxy:
Red Hat Enterprise Linux (RHEL) 8.3
CentOS 8.3
SUSE Linux Enterprise Server (SLES) 15.1

You are currently using Red Hat Enterprise Linux Server 7.9 VSAs.

Development had logged a case for with VMWare who confirmed that 7.9 is not supported.

Are you able to deploy Red Hat Enterprise Linux (RHEL) 8.3 VSAs please?

Now I checked the logs and I don’t think VMWare would help here.

Are you able to deploy Red Hat Enterprise Linux (RHEL) 8.3 VSAs please?

Wow! Great suggestion. Yes, sure I can poke our Linux team and ask them to re-deploy the latest RHEL version they have now. Otherwise I will install CentOS and that’s it. 

 

The only thing I can’t understand is why Synth Full fails due to the last incremental backup? 

Especially considering the fact that all incremental backups are green

From the ticket the engineer said that the logs keep rolling over which is why this is difficult to find the root cause.

Now just to let you know please read below:

CentOS 7 and 8 are the final releases of CentOS Linux.

The end of life dates for CentOS 7 and 8 are as follows: CentOS 8 - December 31, 2021. CentOS 7 - June 30, 2024

I confirmed with SE that we can switch to Windows Server 2022. I am on it.

I don't see Win 2022 being supported, only Win 2019 and 2016:

https://vdc-download.vmware.com/vmwb-repository/dcr-public/ee65cc8c-0701-4b39-a8d3-7025818a02ea/dfaaa64d-f1fc-4722-9b21-cf10fb030eeb/VDDK-700-ReleaseNotes.html#:~:text=VDDK%207.0%20offers%20the%20following,perspectives%20for%20block%2Dbased%20recovery.

https://vdc-repo.vmware.com/vmwb-repository/dcr-public/b677d84a-d0a2-46ab-99bc-590c2f6281b9/d1496e6a-0687-4e13-afdd-339a11e33fab/VDDK-703c-ReleaseNotes.html

and based on https://documentation.commvault.com/2022e/expert/3368_system_requirements_for_virtual_server_agent_with_vmware.html

it is supported

Actually, interesting fact:

Changes and New Features

VDDK 7.0.3.2 offers the following New features:

  • Fix to reduce the number of connections from VDDK to vCenter Server, as requested by backup partners.
  • Corrected HotAdd failures due to mismatch of controller ordering between Linux proxy OS and vSphere, as requested by customers.
  • New configuration options for buffer settings of NFC asynchronous I/O. See resolved issues below.
  • VDDK 7.0.3.2 supports five additional operating systems as backup proxy VMs for HotAdd and other transports: Red Hat Enterprise Linux (RHEL) 7.9, 8.6, 9.0, Ubuntu 18.04, and Windows 2022, in addition to those supported by VDDK 7.0.3.1.

 

Why “

Development had logged a case for with VMWare who confirmed that 7.9 is not supported.

”

We are not on VDDK 7.0.3.2 we are on 7.0.3 and it wasn’t in the last VDDK.

So it looks like they added it now. But you can of course check with VMware and confirm.

As I can see in the TR we have found other error messages and you are working on it.

 

Upd: the issue magically resolved by upgrading from SP28 to SP30. 

Reply


Terms & ConditionsCookie settings