The index is stored with the media of the backup. If you look at contents of a job on media you’ll see data and index. The index the browse references to work it’s way back is stored locally on the media agents and you can see the settings in the catalog tab. If the index no longer exists on the media agent it has to be restored to get to the data you are looking for. Due to a number of factors such as age, media type, amount of objects backed up, an index restore may take some time to complete. 

This should spawn an index restore job in the job controller. If there is not job spawned or the index restore job reports an error you should open a ticket for review. 

Hi @ZachHeise , thanks for coming here first!

There’s a few possibilities, including that the index might be rebuilding already, but taking a while (I often see where the browse on older backups works later in the day upon retrying).

A few quick items to check:

• Is there plenty of room for the restored Index?  I doubt this is the issue if others subclient browses work.
• Can you try using another Media Agent?  
• Can you see if any upgrade was done to the CS or MAs on or around this day?

There *IS* an undocumented registry key that MIGHT work, but it requires support to confirm first.  The key allows the recon to skip older cycles (which is what you can mention if you need to open a support case).

See if the above help, and if not, create a support case and share the number here for tracking 

Hi @Mike Struening and @Aplynx ,

Unfortunately we only have the one mediaagent (mediaagent01) for our oldest 12 month data. We have mediaagent02 in an offsite location but that’s only backing up the last 2 weeks in case of a disaster situation onsite; we’re okay with losing the full-year archive only in case of a disaster (had to cut costs somewhere after all).

Yes, the mediaagents have plenty of room on them.

Unfortunately in the Job Controller there’s no sign than any index rebuilds are being done either (i.e. nothing is being spawned).

How would I look at the “contents of a job on media” to see if there’s both data and index?

We keep the index cache, mediaagent DDB, and jobresults folders as 3 separate folders on the root of a 3TB SSD in mediaagent01. Confirmed that there’s still 900GB of free space on that SSD.

Argh! Okay I tried again to browse our oldest synth full and THIS time it worked in the Browse window. The restore also completed successfully.

Okay, so I guess I should have just been more patient. But it’s weird that it didn’t give me any sort of Job Controller feedback that some rebuild actually was occurring. Is there a log file I can make a note of to look for the status of a rebuild if I need to go back to my oldest indexes?

I did check my Catalog tab settings on MA01 and saw the setting for “index retention” only had “retain index for” set to only 15 days (probably again to save space) unless that’s unrelated.

I’m very relieved to have a successful restore done, I just wish things were less opaque about when commvault needs to do something in the background before an admin can browse.

Glad it's working!

Indexing logs are a bit complex (there’s more than one phase/log).  If it happens again, I would just right-click the job and view logs.  Any relevant logs will show up and allow you to confirm the restore is progressing.

Hi Mike - I would have done that, if a new job would have shown up telling me that an index restore was occurring.

That was my whole point; there was nothing that I could see in the Java console showing me that any sort of action was happening; all I could see was that I wasn’t being allowed to Browse certain synth full backups.

Ah, I’m sorry.  Yeah, so you essentially didn’t even get a job (for say, a restore) to even check.

If you need to do a quick check, start with these, though note you'll see the guid, not the subclient names: 

logmanager.log, indexstate.log, indexserver.log, and there’s also indexrestore.log

those cover the index, though you can also check browse.log on the CS to see what is happening at the top level.

Got it. I verified at 2:59 just now that I was having this issue still on a synthetic full job, same error.

On MediaAgent01 I opened up indexstate.log and saw that a new line was written:

21360 5cc0  05/12 14:59:05 ### DB [2EF7BB31-A4BD-4F02-A349-53455140199B] - Engine [1] - Status [IDX_DB_RECONSTRUCTION] - ReconJobcount [1]

I then waited until I saw the next line:

21360 275c  05/12 15:20:53 ### DB [2EF7BB31-A4BD-4F02-A349-53455140199B] - Engine [1] - Status [IDX_DB_SUCCESS]

And verified that now I can open the synth full job in question.

I will make a note of this for the future that if my support staff see this error, they should wait 20+ minutes and try again. Thanks.

Awesome, and glad to help!!