Solved

CTREE error 652 when trying to backup Nutanix distributed file share

  • 21 July 2022
  • 8 replies
  • 194 views

Badge +2

Hello,

 

I am trying to set up a backup job on a Nutanix file cluster.  There are three shares.  Two are distributed shares and one is standard.  Two f the three back u without issue while the third is giving e CTREE errors

8052  216c  07/19 14:38:25 271 Servant    [---- IMMEDIATE BACKUP REQUEST ----], taskid [115] Clnt[TIDCCFG File Cluster] AppType[Windows File System][33] BkpSet[defaultBackupSet] SubClnt[Production Share] BkpLevel[Full][1]


tidcacs01 \ FileScan.log:

4788 26ec 07/19 16:25:38 271 CTREE rebuild failed with error 652 (duplicate keys rejected and listed); sysiocod=0
4788 26ec 07/19 16:25:38 271 CTREE resumeAndRebuild: unable to rebuild index

4788 26ec 07/19 16:25:38 271 CScanTraverseFolderPrivate::~CScanTraverseFolderPrivate(130) - --- 0:00.000698
4788 26ec 07/19 16:25:38 271 CNutanixScanEngine::ProcessVolumeWorkUnit(268) - ProcessWorkUnit failed, error=0x80070306:{CNutanixScanEngine::ProcessWorkUnit(376)/Cannot perform recursive scan on this work unit due to 0x80070306:{CNutanixScanEngine::ProcessWorkUnitFull(502)} + {CvScanDB::rebuildIndex(367)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Rebuild Index Failed due to unknown error}} + {CNutanixScanEngine::ProcessWorkUnitFull(502)} + {CvScanDB::rebuildIndex(367)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Rebuild Index Failed due to unknown error}
4788 26ec 07/19 16:25:38 271 CVJobClient::setPendingCauseBackend(1068) - EventLevel=9, MessageId=0x1100008E, Flags=0x0, Event=0xE811008E:{CNutanixScanEngine::ProcessVolumeWorkUnit(271)/EventMessage.0x1100008E.17:142.(Failed to process the WorkUnit with error [One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774), Rebuild Index Failed due to unknown error (W32.774)])}
4788 26ec 07/19 16:25:38 271 CNutanixScanEngine::Process(228) - ProcessVolumeWorkUnit failed, error=0x80070306:{CNutanixScanEngine::ProcessVolumeWorkUnit(275)} + {CNutanixScanEngine::ProcessWorkUnit(376)/Cannot perform recursive scan on this work unit due to 0x80070306:{CNutanixScanEngine::ProcessWorkUnitFull(502)} + {CvScanDB::rebuildIndex(367)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Rebuild Index Failed due to unknown error}} +

Not sure what to do here.  I have Nutanix looking at logs and am wondering if it also may be something on the Commvault side of things.

icon

Best answer by gnussbaum 2 September 2022, 20:33

View original

8 replies

Userlevel 2
Badge +7

Please provide the following:

 

Commvault Version:

Nutanix version:

Userlevel 7
Badge +16

Looking at the log it seems to originate from the nutanix file system, but I'm not entirely sure.

Is this a distributed share or the standard one?

This is the initial full correct?

Badge +2

This is a distributed share.

 

Commvault version = 11.24

Nutanix version = NCC Version: 4.3.0

Userlevel 2
Badge +7

@gnussbaum 

It appears that this may be a Nutanix issue. 

Based on the logs provided it appears that there was a duplicate record “error 652” found in the index. The rebuild attempt was unsuccessful possibly due to open file members, but hard to say without all the logs.  

 

 ### Attempted to rebuild and was unsuccessful ###

271 CTREE resumeAndRebuild: unable to rebuild index

 

This error below could be due to open file members that must be closed in order for the rebuild to occur or transient error.

###   (ERROR_ERRORS_ENCOUNTERED.774))-Rebuild Index Failed due to unknown error} ### 

 

Jobmanager.log may have some more details as to way this occurred. 

  • Have you opened a CV case? 
    • If so please provide the case number so we can track
  • Keep us informed on the updates from Nutanix :) 
Badge +2

Hi,

Thanks for the update.  CV case #220719-561.  I also sent logs to Nutanix and am waiting to hear back.

Userlevel 7
Badge +23

Hi @gnussbaum , following up on this one.  I see the case was archived without a solution.

Can you advise as to the status of the issue?

I believe this was a Nutanix issue, though hpoing you have more info.

Badge +2

Hi Mike.  We found that one of the directories was causing the issue.  It was being scanned twice, causing the job to fail.  Nutanix and Commvault developers met on a call and said it was attributed to a race condition that has been resolved.

Userlevel 7
Badge +23

Very interesting.  glad the two were able to come together and solve it for you.

I moved the Best Answer mark to your latest reply as well.

Thanks!!

Reply