File Server backups not aging out

Run the “Data Retention Forecast” report for the Storage Policy in question and see what it says for why the jobs are being retained.  It will give an explanation which should help clarify.

https://documentation.commvault.com/2022e/expert/39786_data_retention_forecast_and_compliance_report_overview.html

Thanks,
Scott
 

Interesting, I get:  

RECONCILE_BACKUP_PENDING

Job is not prunable and will be retained until a synchronization operation runs when a new backup cycle starts. 

To skip jobs from being retained until a synchronization operation runs, disable the Retain jobs in previous cycle until Source Files and backup index are synchronized in latest cycle parameter on the Data Aging tab of the Media Management Configuration dialog box.

Weirdly my LiveSync process is enabled and runs every 15 minutes.

I’m going to guess your Subclient does NOT have “Reconcile Backup” selected?
 

You can either enable “Reconcile Backup” on the Subclient, or disable the option below in the Control Panel to disable this feature globally.
 

This was a feature introduced around SP7, I think.  Subclients created after that point should have the “Reconcile Backup” selected by default.

What is happening is we want to run a “reconcile” process at the beginning of the backup job to compare the previous backup job with the index to look for any discrepancies.  Your jobs are not aging since this reconcile process has not been running.  The short explanation is this reconcile process is run to catch any edge cases where we could miss data.

Preferred solution:  Enable the reconcile option on the Subclient.
Alternate solution:  Disable the option globally in the Control Panel.

Thanks,
Scott
 

Well this is disturbing… My Reconcile Backup option is actually checked:

I’m guessing I have to open another support ticket to find out why it hasn’t been running.

Yes, probably need a support case to find out why they’re not pruning.

I would be interested in the resolution.  Please share once it’s resolved.

Thanks,
Scott
 

Hi @Ken_H 

Under job history, can you select columns option and select “Reconcile Backup”. This will show which jobs ran reconcile backup.

Thanks,

Sparsh

Of all the history, there’s one reconcile backup: on an incremental backup from March 6, 2023.  If they’re supposed to run every 30 days, I should have seen another one by now.

Ticket 230412-632 has been opened to investigate the missing reconcile backup.

@Ken_H 

Just run a new job, and upload FileScan.log & PostOps.log here. I can check immediately and see whether Reconcile Backup is not running or failing for some unforeseen reason.

Thanks,

Sparsh

The recommendation from the analyst handling 230412-632 was to enable reconcile backups on the default subclient even though I’m dealing with the DataDrives subclient.  They also recommended changing the reconcile backup “Every X days from the previous backup job” to match the storage policy which I interpreted as “Change from every 30 days to every 7 days”.  I’ve made both changes (enable reconcile backup on default, reconcile backup every 7 days) but no new reconcile backup have run and old backups still haven’t aged out.  I’ll upload logs as requested.

@SparshGupta, weirdly my CommVault installation does not have a PostOps.log file on the CommServer.

I’ve run an incremental backup, copied FileScan.log to my desktop and edited it to remove entries prior to today so it wouldn’t be so large.  I’ve renamed “FileScan.log” to “fileScan_log.txt” and have attached it to this post as requested.

Hi @Ken_H 

What is the job ID and backupset / subclient name?

Also, the log is needed from the Windows client where the job is running, not the Commserver.

Thanks,

Sparsh

Job ID is 614916, Backupset is defaultBackupSet, subclient is DataDrives.  The FileScan.log file from INF-SRV0 has been renamed to FileScan_log_INF-SRV0.txt and will be uploaded.  Like the CommServer, “PostOps.log” does not exist.

Sorry, I’ve taken a look at the contents of FileScan.log and don’t think it’s appropriate to post on a public forum and have deleted the file from my previous post.  Thanks for your help - I’ll continue to work with CommVault support.

Hi @Ken_H 

Could you please work with support team and upload all logs from CS / MA / Index Server & Clients along with CSDB?

Reconcile Backup is being triggered, however we are getting error from index server “inf-srvp112.apacorp.net” and not able to fetch files from index.

FileScan.log -

11160 1d90  04/13 09:12:33 614916 CPathCacheFlatFileIndexPrivate::LoadFromPreviousBackingFile(451) - PreviousBackingFile=[C:\Program Files\Commvault\ContentStore\iDataAgent\JobResults\CV_JobResults\iDataAgent\FileSystemAgent\2\395\DCInc.cvf]
11160 1918  04/13 09:12:33 614916 IDXBROWSECL Sending init request [<?xml version="1.0" encoding="UTF-8" standalone="no" ?><databrowse_BrowseRequest opType="1" pathType="0"><entity _type_="0" applicationId="33" backupsetId="173" commCellId="2" instanceId="1" subclientId="395"/><paths path="\**\*"/><timeRange fromTime="0" toTime="0"/><options DoNotCacheFilters="0" PreferContentServer="0" adjustPerLastDRTime="0" browseFromCSDB="0" browseFromJobType="0" browseFromLatestJob="0" bulkOperation="0" bypassSubclientConfig="1" caseInsensitive="0" caseManagerDataCopy="0" createRestoreVectors="0" cycleBrowse="0" disableMetadata="0" disableSnapOptimizer="0" doPrediction="0" downloadSystemScan="0" downloadTrueUpStubScanOutput="0" enforceSuperBrowse="0" fetchFileProperties="0" fixPaths="0" folderDelete="0" fsExtentBrowse="0" getACLs="0" hideEmptyExtents="0" imageBrowse="0" includeAgedData="0" includeAllParents="0" includeCheckSum="0" includeEndUserErasedItems="0" includeExtentCRC="0" includeMetadata="0" includeOnlyCommitedJobs="1" includeRunningJobs="0" includeSoftCommittedJobs="0" includeVersions="0" instantSend="0" internalReq="0" isOneTouchRestore="0" machineLiveBrowse="0" nasFilerType="0" nasSynthDirRestore="2" nasSynthRestore="0" nonRecursiveNASRestore="0" onePassRedundancy="0" onePassStubQuery="0" prepareSystemScan="0" pruneFromSource="0" queryAgedArchiveFileData="0" reCreateRestoreVector="0" refCopy="0" refCopyCloudMigration="0" refCopySF="0" relationshipBrowse="0" restoreIndex="1" sendBackListOfAfiles="0" sendOutputToFile="0" sharedFolderBrowse="0" showDeletedFiles="1" showFavoriteAndShareInfo="0" showFolderSize="0" showOnlyDeleted="0" showOnlyNew="0" showThumbnail="0" skipDataPrediction="0" skipIndexRestore="0" skipInvalidMimeClassification="0" skipLatestVersion="0" skipMountPoints="0" skipNasRenamedDirs="0" skipProgressMessage="0" skipSynthFullJobs="0" synthFull="0" tableViewBrowse="0" translateSyncPath="0" translateToVMReq="0" useLatestCycle="0" useUptoDateIndexOnly="0" verifyArchiverQuery="0" viewBackupCopy="0" vsDiskBrowse="0" vsExtentBrowse="0" vsFileBrowse="1" vsListVMs="0" vsLiveSyncBrowse="0" vsVolumeBrowse="0" vsaFileLevelRestore="0"/><mode mode="2"/><advOptions applyStrictACL="0" copyPrecedence="0" datasourceId="0" includePsuedoItems="0" retryLimit="1" retryWaitSecs="300" skipACLChecks="0" skipUptodateCheck="0" stubAsData="0" timeoutSecs="1800" validateAfiles="1" vmCentricClient="0"/><queries queryId="TrueUp" type="0"><dataParam><paging firstNode="0" pageSize="2147483647" skipNode="0"/></dataParam><whereClause connector="0"><criteria field="0"><values val="*"/></criteria></whereClause></queries></databrowse_BrowseRequest>] to CS
11160 1918  04/13 09:12:34 614916 CTrueUpBrowseHandler::GenerateTrueUpDirChange(258) - Sending browse request to generate TrueUp DirChange
11160 1918  04/13 09:12:34 614916 CVOnDemandSvcClient::SubmitTask() - On Demand service CVODS_indexserver_inf-srvp112.apacorp.net_1 launched at host inf-srvp112.apacorp.net*inf-srvp112.apacorp.net*8400*8402. Unique ID is inf-srvp112.apacorp.net1681398169
11160 1918  04/13 09:12:34 614916 CVOnDemandSvcClient::Attach() - Successfully attached to on Demand service CVODS_indexserver_inf-srvp112.apacorp.net_1 launched at host inf-srvp112.apacorp.net*inf-srvp112.apacorp.net*8400*8402. Unique ID is inf-srvp112.apacorp.net1681398169
11160 1918  04/13 09:12:34 614916 CTrueUpIndexContentCallback::processResp(229) - Browse response returned error; Fail TrueUp DirChange processing
11160 1918  04/13 09:12:34 614916 IDXBROWSECL Callback failed while processing browse error
11160 1918  04/13 09:12:34 614916 CTrueUpBrowseHandler::GenerateTrueUpDirChange(149) - --- 0:01.248896 
11160 1d90  04/13 09:12:37 614916 CPathCacheFlatFileIndexPrivate::LoadFromPreviousBackingFile(431) - --- 0:03.675086 

Thanks,

Sparsh

Update from CommVault support:   

...we need to wait for the next incremental after the Synthetic Full, as this will be the Reconcile backup job. Once the incremental completes on Monday, please try running Data Aging and see if the jobs are still present.

April 18:  No new reconcile backup and none of the backups with a Retain Until date before today’s date have aged out.

Hi @Ken_H 

I will ask support team to get all the required details & escalate the case to Engineering team.

Thanks,

Sparsh

I’ve heard back on ticket 230412-632.  Apparently the True Up (reconcile) backups are failing because of too many Content Indexing requests. as luck would have it, I already have a ticket open for that:  230406-659.

What we did today to try and resolve this was download, install, and run a workflow named “Enable Subclient Indexing”.  The workflow ran into an error but despite that, it did manage to configure an “Indexing” tab (under Properties > Advanced).  Hopefully the incremental backups following this weekend’s synthetic full will be able to do a reconcile backup.  

More to come next week.