Solved

Cleaning up old Index files off of NAS storage

  • 28 November 2022
  • 6 replies
  • 421 views

Userlevel 2
Badge +10

Background:

We added a new NAS storage array to CommVault, and marked the old ones mount points ‘Disabled for Write’. Time has passed and data has aged off, but the only thing left on the old storage (seen via right click on the storage mount path in the CV Java UI, choosing “View Contents”) is a handful of Index related data (with names of “<storage_policy>_IndexServer”, and from iDataAgent “Big Data Apps”) that has data retention dates “Retain Until” from last year back to 2019 ... but it will not age off.

I asked support about it while cleaning up some other CommVault data on these old filers and they indicated that if we upgraded to the “latest greatest” (we’re on 11.24.56), then these index files are treated differently in a later version and will all clean up after the upgrade.

Poking around in the forums, I find there’s an “Index Server Load” report in the web UI with a “Run load balance” feature (Load Balance Index Servers: Precautions, impact to running | Community (commvault.com)…and the report indicates in my environment “You can run load balance to distribute 2 index databases and save 847 MB”

A few questions: 

  1. Does anyone know the version/hotfix I would need to update to that would likely be “the version where the index files are treated differently”? I know I could just update to the latest, I was just wondering if I had to.
  2. Is it possible  the “Run load balance” feature in the “Index Server Load” report might move some of these indexes to the new storage, then they will age off?
  3. Is there a way to track down what’s keeping the each of the indexes from ageing off?  previously we had a lot of data not ageing off because (for example) subclients were turned off and there were incrementals left behind, but for those we could track down the subclient and investigate/decommision completely/etc. and it aged off.
icon

Best answer by Zubair Ahmed Sharief 2 December 2022, 17:51

View original

6 replies

Userlevel 5
Badge +13

Hello @tigger2

To answer your questions:

  1. I am not aware of any “fix” that will age off old indexes. The rule of thumb is that once the data requiring the index ages, the index will age.
  2. The Index Load balance workflow would not age data
  3. You can try checking the Data Retention Forecast and Compliance report for these index backup Job IDs. This might give some clue as to why they haven’t aged.

Data Retention Forecast and Compliance Report Overview - https://documentation.commvault.com/11.26/expert/39786_data_retention_forecast_and_compliance_report_overview.html

 

Thank you,
Collin

Userlevel 2
Badge +10

I checked the Data Retention report, and it appears that:

 

  1. The job ID is “required” by… itself. It is like it’s causing itself to be retained?
  2. The Reason for the retention is “INDEXING_JOB”, which the docs/report links to https://documentation.commvault.com/v11/expert/92408_backupset_level_indexing_index_cache_cleanup.html . Reading it, It seems? that maybe index jobs “roll off” when new ones come in (over time) … but I have these indexing jobs for all of my storage policies on my old storage (that cannot be written to, only read/deleted from, per commvault settings we changed so the commvault would clean up the storage over time as it aged data off). For some reason they are hanging around from 3-1 years ago so I would imagine they would have aged off by now. I’m not seeing anything obvious to tell me “what other jobs” or anything are keeping them there that I could clean up.
Userlevel 3
Badge +9

Hello @tigger2 

Could you try the below please

check the following parameter value "Delete archive files of indexing backup associated with deleted backup sets" under Control Panel >> Media Management Configuration >> Data Aging tab.

If it is set to 0, update it to 1. Run a data aging job afterwards. if the Backup set does not exist, old index will be pruned.

Userlevel 3
Badge +9

Hello @tigger2 

Could you check the below link
Index backup aging | Community (commvault.com)

It looks like we have Diag for this issue for 11.24.52 (UpdateBundle_Build1108123_Form5176) and Rolled it up in MR 11.24.65

Userlevel 2
Badge +10

I set the "Delete archive files of indexing backup associated with deleted backup sets" and ran data ageing and… all the old index files on the old storage are gone! Thanks!

I’m reading the second link you sent right now.

Follow up questions, if that’s ok:

  • It would seem this setting I changed could be left “on” and not cause any issues (that I can see)? I’m not sure why it would default to 0, especially since… why keep indexes of old, deleted backup sets, and also when it seems that index backups “continually refresh” and prune/update themselves. 
  • It seems that the second link you sent talks about how this is a known issue... but maybe it's an issue/bug when this setting does not work?
Userlevel 3
Badge +9

Hello @tigger2 

Follow-up Answers

  1. I do not see any issues either however i would prefer to switch it off after the issue is resolved.
  2. I would recommend upgrading CV to 11.24.65 (i.e., one without any settings or extra config in place)

Reply