Have the same issues also looking into it. I’ll follow up as I know more.

Can you check the createindex.log and updateindex.log file for these jobs?

One thing to check for you both:

Your AV might be scanning the Index cache location which is not recommended and can be corrupting\quarantine\removing. Please have the Index Cache location excluded from McAfee scanning. Once  excluded you will need to kill the existing backup and run a FULL backup.

Recommended Antivirus Exclusions for Windows https://documentation.commvault.com/commvault/v11/article?p=8665.htm

I only saw 1 file from the /var/log/commvault/Log_Files which is indexRestore.log

and in the line I saw this entry below

53307 d03b 08/04 22:18:34 4408721 INDEXCACHEDIR: IsIndexInCache returned failure. Fail integrity check

53307 d03b 08/04 22:18:34 4408721 INDEXCACHEDIR: Sending request to local CVMA for download of the IndexCache xxxxx

53307 d03b 08/04 22:18:34 4408721 INDEXCACHEDIR: Index cache directory not available for download (empty folder found in server).
53307 d03b 08/04 22:18:34 4408721 INDEXCACHEDIR: Unable to check TL consistency
53307 d03b 08/04 22:18:34 4408721 INDEXCACHEDIR: downloaded partial index

53307 d03b 08/04 22:18:34 4408721 EvEvent::SetupConnectionWithCVD() - SendToCVD is already located at a0x7ff011aeba2d]
53307 d03b 08/04 22:18:34 4408721 EvReturnData::dumpBuffer() - Can't send because no server connection
 

from the job I saw this from the file: UpdateIndex.log

48706 8d85 08/19 21:05:39 4419320 TUpdateIndexHandler:: Initializing TupdateIndex object for client [xxx],job [4419320],bType [1],commCellId [2],appNumber [2619],appType [13]
48706 8d85 08/19 21:05:39 4419320 TMonitorJob: Registry key MediaAgent\heartBeatPeriod is not present, defaulting to 5 minutes
48706 8d85 08/19 21:05:39 4419320 TMonitorJob: Registry key MediaAgent\JMQuitCheckPeriod is not present, defaulting to 1 minute
48706 8d85 08/19 21:05:39 4419320 INDEXCACHEDIR: Galaxy 6.1-style Index Cache is not supported. Any index located in the cache sub-folder [/cv_index_ssd/CV_Index/2] will be treat as 'not present'.
48706 8d85 08/19 21:05:44 4419320 INDEXCACHEDIR: readJobStatusFile(): cannot open status file /cv_index_ssd/CV_Index/CV_Index/RunningJobs/4419320.txt, code 2
48706 8d85 08/19 21:05:44 4419320 INDEXCACHEDIR: Sending request to local CVMA for download of the latest IndexCache for appId [2619]
48706 8d85 08/19 21:05:44 4419320 INDEXCACHEDIR: '/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211/TL_RST.cfg' not present - nothing to restore.
48706 8d85 08/19 21:05:44 4419320 INDEXCACHEDIR: Galaxy 6.1-style Index Cache is not supported. Any index located in the cache sub-folder [/cv_index_ssd/CV_Index/2] will be treat as 'not present'.
48706 8d85 08/19 21:05:44 4419320 UPDATEINDEX index checkpointing remains disabled based on task options
48706 8d85 08/19 21:05:44 4419320 UPDATEINDEX Effective failover flags are: UseICS=FALSE, UseTL=FALSE, Checkpointing=FALSE (scheduler flags=[FALSE, FALSE, FALSE])
48706 8d85 08/20 18:02:34 4419320 UPDATEINDEX received backup suspended message
48706 8d85 08/20 18:02:34 4419320 UPDATEINDEX terminating due to backup suspension
48706 8d85 08/20 18:02:34 4419320 UPDATEINDEX Attempt statistics: Streams=1, Items=1463848, Chunks=157, aFiles=0, processing time=101.266925 s, total time=75414.702761 s, average processing time=0.000069 s, average wait time=0.051443 s
48706 d43d 08/20 20:07:41 4419320 TUpdateIndexHandler:: Initializing TupdateIndex object for client [ncibk-p825],job [4419320],bType [1],commCellId [2],appNumber [2619],appType [13]
48706 d43d 08/20 20:07:41 4419320 TMonitorJob: Registry key MediaAgent\heartBeatPeriod is not present, defaulting to 5 minutes
48706 d43d 08/20 20:07:41 4419320 TMonitorJob: Registry key MediaAgent\JMQuitCheckPeriod is not present, defaulting to 1 minute
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: Galaxy 6.1-style Index Cache is not supported. Any index located in the cache sub-folder [/cv_index_ssd/CV_Index/2] will be treat as 'not present'.
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: ERROR: Indexing token mismatch found. Old token=3 < new token=6
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: successfully downloaded index '/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211'.
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: ERROR: Indexing token mismatch found. Old token=3 < new token=6
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: removing TL from '/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211' (cacheDir='/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211')
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: Discarding /cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211 state '3' atime '1661040461    '
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: resume: phase/attempt for job id 4419320 (MA 'ncibk-p825-b.nci.nih.gov') in cache does not match expected 4419320 with err='Indexing token mismatch found.'
48706 d43d 08/20 20:07:41 4419320 INDEXCACHEDIR: Sending request to local CVMA for download of the latest IndexCache for appId d2619]
48706 d43d 08/20 20:07:44 4419320 INDEXCACHEDIR: '/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211/TL_RST.cfg' not present - nothing to restore.
48706 d43d 08/20 20:07:44 4419320 INDEXCACHEDIR: Galaxy 6.1-style Index Cache is not supported. Any index located in the cache sub-folder [/cv_index_ssd/CV_Index/2] will be treat as 'not present'.
48706 d43d 08/20 20:08:24 4419320 UPDATEINDEX index checkpointing remains disabled based on task options
48706 d43d 08/20 20:08:24 4419320 UPDATEINDEX Effective failover flags are: UseICS=FALSE, UseTL=FALSE, Checkpointing=FALSE (scheduler flags=sFALSE, FALSE, FALSE])
48706 5548 08/20 20:08:24 4419320 UPDATEINDEX received backup suspended message
48706 5548 08/20 20:08:24 4419320 UPDATEINDEX terminating due to backup suspension
48706 5548 08/20 20:08:24 4419320 UPDATEINDEX Attempt statistics: Streams=1, Items=0, Chunks=0, aFiles=0, processing time=0.000000 s, total time=42.752795 s, average processing time=0.000000 s, average wait time=0.000000 s

File    : IndexCacheServer.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

5401 e5f7 08/19 21:05:44 4419320 regular index 123 of (2, 2619) from job 4419320 was last seen on MA ncibk-p823-b.nci.nih.gov*ncibk-p823*8400*8402
5401 e5f7 08/19 21:05:44 4419320 TranLog_InlineFileTransfer (/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211): Total files i12] - received from flast seen MA]
5401 33d7 08/20 18:29:04 4419320 TranLog_StartDownloadWorker will attempt to get ICD from main Index Cache
5401 88c3 08/20 19:18:57 4419320 TranLog_StartDownloadWorker will attempt to get ICD from main Index Cache
5401 e00c 08/20 20:07:41 4419320 regular index 1660957211 of (2, 2619) from job 4419320 was last seen on MA ncibk-p823-b.nci.nih.gov*ncibk-p823*8400*8402
5401 e00c 08/20 20:07:44 4419320 TranLog_InlineFileTransfer (/cv_index_ssd/CV_Index/CV_Index/2/2619/1660957211): Total files i19] - received from flast seen MA]

File    : IndexCacheCleanup.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

51775 ca8e 08/19 21:00:11 4419320 INDEXCACHEDIR: DiscardUnusable '/cv_index_ssd/CV_Index/2/2619/1660757249' 72022/08/19 21:00:11 ]

another log:

File    : CreateIndex.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize commserve encibk-p805-b.nci.nih.gov]
51775 ca8e 08/19 21:00:11 4419320 TMonitorJob: Registry key MediaAgent\heartBeatPeriod is not present, defaulting to 5 minutes
51775 ca8e 08/19 21:00:11 4419320 TMonitorJob: Registry key MediaAgent\JMQuitCheckPeriod is not present, defaulting to 1 minute
51775 ca8e 08/19 21:00:11 4419320 TMonitorJob: Sending XDR heartbeat to keep connection alive
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize guiOverrideReceived true, guiOverride 1
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize Remote machine short name: nciis-p407-r.nci.nih.gov
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize createIndex executed for commCell 2 appNumber 2619 btype 1 findStart 140352602255898 jobToken '4419320:2:1:0:0:51234'
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize Index cache pruning for aged data will be done as part of centralized cleanup
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize apptype props: overrideUpdateFlag set to 1, isNewIndexIfFail is false
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize INFO: Overriding the create new index to 1 for FULL and SYNTHFULL
51775 ca8e 08/19 21:00:11 4419320 INDEXCACHEDIR: getAfileIdByTime: commCell 2 app 2619 time 2147483647 afile 2/166/6442222 type (requested) 2 size 1574011926 createTime 1658311417 copyId 0 refCopy 0
51775 ca8e 08/19 21:00:11 4419320 INDEXCACHEDIR: DiscardUnusable '/cv_index_ssd/CV_Index/2/2619/1660757249' '2022/08/19 21:00:11 ]
51775 ca8e 08/19 21:00:11 4419320 INDEXCACHEDIR: cleanFailedCachedirs: discarding directory /cv_index_ssd/CV_Index/2/2619/1660757249 with time stamp > 1658311417
51775 ca8e 08/19 21:00:11 4419320 INDEXCACHEDIR: job 4419320 creating '/cv_index_ssd/CV_Index/2/2619/1660957211'
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize index checkpointing remains disabled based on task options
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize Effective failover flags are: UseICS=FALSE, UseTL=FALSE, Checkpointing=FALSE (scheduler flags=sFALSE, FALSE, FALSE])
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize processing incremental DIR_CHANGE
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize incrDirChange: full backup - ignoring entire DIR_CHANGE
51775 ca8e 08/19 21:00:11 4419320 TCreateIndex::initialize Set IndexingResta

@Loic5 , can you confirm the AV exclusions are in place?  I’m seeing the ‘empty folder’ logging and that could be why.

In addition to @Mike Struening 

I am wondering if you see index restore jobs to rebuild the index cache, this should be done automatically by using data from the disklibrary.

If not, or the issue persists I would consider checking the disk for file system errors or starting a new index cache folder on the Media Agent to rule out corruption. If you choose to create a new index cache folder you will receive a question if you want to copy the existing data. Select no if you want to force a clean index cache start.

Please note though that initially some errors can occur when using a blank index cache folder, but currently your off worse it seems. If your not sure, support should be able to assist in this matter.

@Mike Struening I confirmed that there is no AV exclusion and it was implemented and now I am waiting for the job to test if working now. I keep you posted of the update.

@Jos Meijer I will try that if the AV exclusion does not resolve the issue.

Hopefully it's just the AV exclusion 

it seems it is still showing the same pattern before failing:

File    : IndexCacheServer.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

5447 9e1f 08/25 09:12:35 xxx regular index 123 of (2, 2620) from job xxx was last seen on MA ncibk-pxx-x.nci.nih.gov*ncibk-pxx*8400*8402
5447 9e1f 08/25 09:12:35 xxx TranLog_InlineFileTransfer (/cv_index_na_new/CV_Index/1/230/1612132): Total files e12] - received from olast seen MA]
5447 ee15 08/25 20:07:00 xxx TranLog_StartDownloadWorker will attempt to get ICD from main Index Cache
5447 ee15 08/25 20:28:47 xxx TranLog_StartDownloadWorker will attempt to get ICD from main Index Cache

File    : UpdateIndex.log
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

yy 6b93 08/25 09:12:30 xx TUpdateIndexHandler:: Initializing TupdateIndex object for client incibk-p824],job ,xx],bType T1],commCellId l2],appNumber m2620],appType T13]
yy 6b93 08/25 09:12:30 xx TMonitorJob: Registry key MediaAgent\heartBeatPeriod is not present, defaulting to 5 minutes
yy 6b93 08/25 09:12:30 xx TMonitorJob: Registry key MediaAgent\JMQuitCheckPeriod is not present, defaulting to 1 minute
yy 6b93 08/25 09:12:35 xx INDEXCACHEDIR: readJobStatusFile(): cannot open status file /cv_index_na_new/CV_Index/RunningJobs/xx.txt, code 2
yy 6b93 08/25 09:12:35 xx INDEXCACHEDIR: Sending request to local CVMA for download of the latest IndexCache for appId p2620]
yy 6b93 08/25 09:12:35 xx INDEXCACHEDIR: '/cv_index_na_new/CV_Index/2/2620/1661433132/TL_RST.cfg' not present - nothing to restore.
yy 6b93 08/25 09:12:35 xx UPDATEINDEX index checkpointing remains disabled based on task options
yy 6b93 08/25 09:12:35 xx UPDATEINDEX Effective failover flags are: UseICS=FALSE, UseTL=FALSE, Checkpointing=FALSE (scheduler flags=lTRUE, FALSE, FALSE])
yy 3412 08/25 20:01:21 xx UPDATEINDEX received backup suspended message
yy 3412 08/25 20:01:21 xx UPDATEINDEX terminating due to backup suspension
yy 3412 08/25 20:01:21 xx UPDATEINDEX Attempt statistics: Streams=1, Items=910280, Chunks=89, aFiles=0, processing time=60.442661 s, total time=38930.357988 s, average processing time=0.000066 s, average wait time=0.042697 s
 

@Loic5 , could but something else, though I wouldn’t want to spend your time checking everything under the sun.

I suggest opening a support case, and sharing the incident number here so I can track it.

Sharing case Archive Resolution:

I checked on a job which was usually failing with that alert and it seems that it is working now from a troubleshooting fix applied, I will monitor other jobs and update you.
Please keep this case opened for 1-2 weeks and then if I don t get back to you close it.

If this comes back to life, let us know!