+15
Hello everyone, I just noticed my file server backups are not aging out following the upgrade to 11.30.32. My file server has the CommVault client installed and I do incremental backups throughout the week with synthetic full backups on Friday night. The last synthetic full backup of the month is retained for a year and the last Syn-Full of the year is retained for 3 years. Does anyone have suggestions on what to check?
Best answer by Ken_H
View original
+18
Run the “Data Retention Forecast” report for the Storage Policy in question and see what it says for why the jobs are being retained. It will give an explanation which should help clarify.
https://documentation.commvault.com/2022e/expert/39786_data_retention_forecast_and_compliance_report_overview.html
Thanks,
Scott
+15
Interesting, I get:
RECONCILE_BACKUP_PENDING
Job is not prunable and will be retained until a synchronization operation runs when a new backup cycle starts.
To skip jobs from being retained until a synchronization operation runs, disable the Retain jobs in previous cycle until Source Files and backup index are synchronized in latest cycle parameter on the Data Aging tab of the Media Management Configuration dialog box.
+15
Weirdly my LiveSync process is enabled and runs every 15 minutes.
+18
I’m going to guess your Subclient does NOT have “Reconcile Backup” selected?
You can either enable “Reconcile Backup” on the Subclient, or disable the option below in the Control Panel to disable this feature globally.
This was a feature introduced around SP7, I think. Subclients created after that point should have the “Reconcile Backup” selected by default.
What is happening is we want to run a “reconcile” process at the beginning of the backup job to compare the previous backup job with the index to look for any discrepancies. Your jobs are not aging since this reconcile process has not been running. The short explanation is this reconcile process is run to catch any edge cases where we could miss data.
Preferred solution: Enable the reconcile option on the Subclient.
Alternate solution: Disable the option globally in the Control Panel.
Thanks,
Scott
+15
Well this is disturbing… My Reconcile Backup option is actually checked:
I’m guessing I have to open another support ticket to find out why it hasn’t been running.
+18
Yes, probably need a support case to find out why they’re not pruning.
I would be interested in the resolution. Please share once it’s resolved.
Thanks,
Scott
+10
Hi
Under job history, can you select columns option and select “Reconcile Backup”. This will show which jobs ran reconcile backup.
Thanks,
Sparsh
+15
Of all the history, there’s one reconcile backup: on an incremental backup from March 6, 2023. If they’re supposed to run every 30 days, I should have seen another one by now.
+15
Ticket 230412-632 has been opened to investigate the missing reconcile backup.
+10
Just run a new job, and upload FileScan.log & PostOps.log here. I can check immediately and see whether Reconcile Backup is not running or failing for some unforeseen reason.
Thanks,
Sparsh
+15
The recommendation from the analyst handling 230412-632 was to enable reconcile backups on the default subclient even though I’m dealing with the DataDrives subclient. They also recommended changing the reconcile backup “Every X days from the previous backup job” to match the storage policy which I interpreted as “Change from every 30 days to every 7 days”. I’ve made both changes (enable reconcile backup on default, reconcile backup every 7 days) but no new reconcile backup have run and old backups still haven’t aged out. I’ll upload logs as requested.
+15
I’ve run an incremental backup, copied FileScan.log to my desktop and edited it to remove entries prior to today so it wouldn’t be so large. I’ve renamed “FileScan.log” to “fileScan_log.txt” and have attached it to this post as requested.
+10
Hi
What is the job ID and backupset / subclient name?
Also, the log is needed from the Windows client where the job is running, not the Commserver.
Thanks,
Sparsh
+15
Job ID is 614916, Backupset is defaultBackupSet, subclient is DataDrives. The FileScan.log file from INF-SRV0 has been renamed to FileScan_log_INF-SRV0.txt and will be uploaded. Like the CommServer, “PostOps.log” does not exist.
+15
Sorry, I’ve taken a look at the contents of FileScan.log and don’t think it’s appropriate to post on a public forum and have deleted the file from my previous post. Thanks for your help - I’ll continue to work with CommVault support.
+10
Hi
Could you please work with support team and upload all logs from CS / MA / Index Server & Clients along with CSDB?
Reconcile Backup is being triggered, however we are getting error from index server “inf-srvp112.apacorp.net” and not able to fetch files from index.
FileScan.log -
11160 1d90 04/13 09:12:33 614916 CPathCacheFlatFileIndexPrivate::LoadFromPreviousBackingFile(451) - PreviousBackingFile=[C:\Program Files\Commvault\ContentStore\iDataAgent\JobResults\CV_JobResults\iDataAgent\FileSystemAgent\2\395\DCInc.cvf]
11160 1918 04/13 09:12:33 614916 IDXBROWSECL Sending init request [<?xml version="1.0" encoding="UTF-8" standalone="no" ?><databrowse_BrowseRequest opType="1" pathType="0"><entity _type_="0" applicationId="33" backupsetId="173" commCellId="2" instanceId="1" subclientId="395"/><paths path="\**\*"/><timeRange fromTime="0" toTime="0"/><options DoNotCacheFilters="0" PreferContentServer="0" adjustPerLastDRTime="0" browseFromCSDB="0" browseFromJobType="0" browseFromLatestJob="0" bulkOperation="0" bypassSubclientConfig="1" caseInsensitive="0" caseManagerDataCopy="0" createRestoreVectors="0" cycleBrowse="0" disableMetadata="0" disableSnapOptimizer="0" doPrediction="0" downloadSystemScan="0" downloadTrueUpStubScanOutput="0" enforceSuperBrowse="0" fetchFileProperties="0" fixPaths="0" folderDelete="0" fsExtentBrowse="0" getACLs="0" hideEmptyExtents="0" imageBrowse="0" includeAgedData="0" includeAllParents="0" includeCheckSum="0" includeEndUserErasedItems="0" includeExtentCRC="0" includeMetadata="0" includeOnlyCommitedJobs="1" includeRunningJobs="0" includeSoftCommittedJobs="0" includeVersions="0" instantSend="0" internalReq="0" isOneTouchRestore="0" machineLiveBrowse="0" nasFilerType="0" nasSynthDirRestore="2" nasSynthRestore="0" nonRecursiveNASRestore="0" onePassRedundancy="0" onePassStubQuery="0" prepareSystemScan="0" pruneFromSource="0" queryAgedArchiveFileData="0" reCreateRestoreVector="0" refCopy="0" refCopyCloudMigration="0" refCopySF="0" relationshipBrowse="0" restoreIndex="1" sendBackListOfAfiles="0" sendOutputToFile="0" sharedFolderBrowse="0" showDeletedFiles="1" showFavoriteAndShareInfo="0" showFolderSize="0" showOnlyDeleted="0" showOnlyNew="0" showThumbnail="0" skipDataPrediction="0" skipIndexRestore="0" skipInvalidMimeClassification="0" skipLatestVersion="0" skipMountPoints="0" skipNasRenamedDirs="0" skipProgressMessage="0" skipSynthFullJobs="0" synthFull="0" tableViewBrowse="0" translateSyncPath="0" translateToVMReq="0" useLatestCycle="0" useUptoDateIndexOnly="0" verifyArchiverQuery="0" viewBackupCopy="0" vsDiskBrowse="0" vsExtentBrowse="0" vsFileBrowse="1" vsListVMs="0" vsLiveSyncBrowse="0" vsVolumeBrowse="0" vsaFileLevelRestore="0"/><mode mode="2"/><advOptions applyStrictACL="0" copyPrecedence="0" datasourceId="0" includePsuedoItems="0" retryLimit="1" retryWaitSecs="300" skipACLChecks="0" skipUptodateCheck="0" stubAsData="0" timeoutSecs="1800" validateAfiles="1" vmCentricClient="0"/><queries queryId="TrueUp" type="0"><dataParam><paging firstNode="0" pageSize="2147483647" skipNode="0"/></dataParam><whereClause connector="0"><criteria field="0"><values val="*"/></criteria></whereClause></queries></databrowse_BrowseRequest>] to CS
11160 1918 04/13 09:12:34 614916 CTrueUpBrowseHandler::GenerateTrueUpDirChange(258) - Sending browse request to generate TrueUp DirChange
11160 1918 04/13 09:12:34 614916 CVOnDemandSvcClient::SubmitTask() - On Demand service CVODS_indexserver_inf-srvp112.apacorp.net_1 launched at host inf-srvp112.apacorp.net*inf-srvp112.apacorp.net*8400*8402. Unique ID is inf-srvp112.apacorp.net1681398169
11160 1918 04/13 09:12:34 614916 CVOnDemandSvcClient::Attach() - Successfully attached to on Demand service CVODS_indexserver_inf-srvp112.apacorp.net_1 launched at host inf-srvp112.apacorp.net*inf-srvp112.apacorp.net*8400*8402. Unique ID is inf-srvp112.apacorp.net1681398169
11160 1918 04/13 09:12:34 614916 CTrueUpIndexContentCallback::processResp(229) - Browse response returned error; Fail TrueUp DirChange processing
11160 1918 04/13 09:12:34 614916 IDXBROWSECL Callback failed while processing browse error
11160 1918 04/13 09:12:34 614916 CTrueUpBrowseHandler::GenerateTrueUpDirChange(149) - --- 0:01.248896
11160 1d90 04/13 09:12:37 614916 CPathCacheFlatFileIndexPrivate::LoadFromPreviousBackingFile(431) - --- 0:03.675086
Thanks,
Sparsh
+15
Update from CommVault support:
...we need to wait for the next incremental after the Synthetic Full, as this will be the Reconcile backup job. Once the incremental completes on Monday, please try running Data Aging and see if the jobs are still present.
+15
April 18: No new reconcile backup and none of the backups with a Retain Until date before today’s date have aged out.
+10
Hi
I will ask support team to get all the required details & escalate the case to Engineering team.
Thanks,
Sparsh
+15
I’ve heard back on ticket 230412-632. Apparently the True Up (reconcile) backups are failing because of too many Content Indexing requests. as luck would have it, I already have a ticket open for that: 230406-659.
What we did today to try and resolve this was download, install, and run a workflow named “Enable Subclient Indexing”. The workflow ran into an error but despite that, it did manage to configure an “Indexing” tab (under Properties > Advanced). Hopefully the incremental backups following this weekend’s synthetic full will be able to do a reconcile backup.
More to come next week.
+15
Hello everyone,
As noted above, I had a session with CommVault support to download, install, and run the “Enable Subclient Indexing” workflow. The incremental backup that ran that same evening ran an reconcile backup - I just didn’t notice until today (four days later). All the old backups with Retain Until dates that were several weeks in the past have aged out and I have emailed that the support ticket can be closed.
Thanks to everyone for the support on this. I don’t believe that running the workflow is a generic solution for backups not aging out so for anyone else who’s in a similar situation, I’d say the solution is to open a ticket with CommVault Technical Support.
Ken
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
