Determining which clients are causing the most index growth?

Hi Onno - I have heard from my linux file server admin that we have a few new research projects that involve web scraping into individual JSON files. This is still assumptions on my part, but I believe they use R to analyze trends and patterns in the fields of each JSON - twitter usage analysis I know is a big one, although that one has been going on for years, not months.

So they are periodically using their R programs to ingest new loads of tens of thousands of JSON files, each one mere kilobytes in size, to then output an analysis. They’ll then delete that week’s folder of say, 20,000 files - that might not be more than 20-30MB in size, and start collecting a new folder to repeat the process.

If that pattern of usage holds true, and they’re doing this on my file server, is it reasonable to assume that commvault’s index, based on my fairly generous retention policy on my primary mediaagent, would grow like this?

My “data written” values are not that large. Like I said, I don’t think large files are being written. I think many, very small files are being written (and then deleted after a few days).

I’ll look around if there’s a report someone made that can tell me this.

(my offsite, DR-only mediaagent02 which only has a retention of one week, for example, does NOT have this same problem with running out of space on its DDB/index drive!)

There is an “Indexing Status” report available at the software store.  This report gives each client, what agents are installed and the index version for each:

https://cloud.commvault.com/webconsole/softwarestore/#!/135/663/11353

Is it possible that report you’re showing me is incorrect, or just not showing me the full picture? According to the most up to date report on my command center, all my clients with index values listed there are only equal to about 340GB - I’d love it if my index folder size was that small.

Yes - that report is only for V2 indexing I think. There could still be indexes on disk for V1 clients. Clients could be on V2 or V1 depending on when they were deployed in the V11 lifecycle. 

Might be worth getting a tool like treesize and run it over the directly to see if there are any other whales to investigate.

Okay, I’ve successfully converted all my file servers to use subclient indexes! Haven’t gained any space back on the D:\IndexCache folder yet though. Is there a command I can run to force commvault to remove the client-level indexes that are no longer in use? perhaps they’d be cleaned up ‘eventually’ but I’d like to get space back sooner rather than later - if I can!

You can run the Load Balance Report for the index: https://documentation.commvault.com/11.24/expert/100883_index_server_load_overview.html

If the media agents have overlapping data paths, you should be able to run the load balance workflow: https://documentation.commvault.com/11.24/expert/100974_running_index_server_load_balancing_with_load_balance_index_servers_workflow_indexing_version_2_commcell_console.html

I’d also recommend reviewing the space available to confirm you have the proper space available.
 

Hi @Aplynx I don’t think that can apply to us because our two mediaagents are purposely separated - completely different data paths since MA01 is onsite and for primary recoveries, MA02 is offsite and only for DR. That ‘load balance workflow’ you mention is already running automatically every week as part of the commvault-created default schedule, but since no data paths are shared between the two media agents, and I don’t have any other on-prem index servers, it probably isn’t doing anything (i.e. each job in the history finishes after 5 seconds).

I downloaded the workflow you mentioned and ran it, and it points out (accurately) that within the index, 750GB is consumed by ‘index size’ and 570GB is consumed by ‘logs size’ which matches what Treesize on "D:\IndexCache\CvIdxDB" and "D:\IndexCache\CvIdxLogs" says, respectively.

The number of files on this file server has more than doubled since May, and the total size of the backup (based on looking at the weekly synthetic fulls) has almost doubled. I knew when we added these new projects to the file server, that we’d see some index growth, but the amount of disk space its using has now tripled, which was very surprising.

I’d also recommend reviewing the space available to confirm you have the proper space available.

Right - if it turns out that doubling the size of this file server has resulted, accurately, in the index needing to triple in size (which is what it has done) in order for commvault to function, then so be it, i’ll need to add more SSDs.

I am still holding out hope though, that it is still possible for me to reclaim space by removing the now-unneeded backupsetindexes and I won’t need to buy new SSDs.

I’ve created ticket 220801-662 @Mike Struening 

Got off the phone with commvault support earlier today, he directed my attention to this article: https://documentation.commvault.com/11.26/expert/144770_modifying_index_pruning_settings_for_subclient_level_indexes.html

I was really hoping to only save a couple days worth of index results for this linux subclient and therefore selected the “retain index in cache for # of days” - and only picked 2 days.

I was very surprised and disappointed to see the line “(note that a minimum of 2 cycles will be retained, no matter what number you enter in this field).” for that option. That completely defeats the purpose for me if choosing the day option is going to force this subclient to hold onto 2 cycles worth (about 2 months in our case) worth of index data.

In the end, I selected the “Retain index in cache for [Number] cycles.” and just put in 1 cycle. It’s still way longer than I want this cache to be around. But hopefully it will slow the bleeding.

I’d love to see, in a future version, the ability to really use that option to only save index cache for [Number] of days.

“ I checked internally and Development added that by default to prevent data loss, that is no to the CMR “

long shot, no luck.

I get that commvault shouldn’t have this as an easily accessible option to protect junior admins from shooting themselves in the foot, but it’s disappointing that there’s no way to take off the training wheels here.

Hi again folks - so using some scripting to search for new OR changed files on the linux file server every 6 hours, corresponding with our commvault backup schedule, we’ve found a user who is creating 85,000 files every day, JSONs as expected.

Is there any way to interpret this quantity of file creation/modification/deletion as to how it would affect the commvault index? How much should the index folder on a mediaagent grow per modified file record on a client?

On my side, it looks like we’ve got 150-200GB less on the Index drive every week vs the previous week.

Right, that’s what I figured - part of that same automated scraping script that user is doing is also generating 10 400MB stata files every night too. But I don’t care about 10 files being made and neither does the index - right?

I’m much more concerned about the 84,990 files that are 10-300KB in size being generated at the same time, and I was just wondering if that number of files, created every day (and then deleted the next day) would be a reasonable thing to point at and say “ah-hah, that’s why our index is growing so quickly and exponentially”

Thanks, Mike!