commvault.com commvault.com
Solved

Unable to restore from my oldest synth fulls, tells me index is being rebuilt?

ZachHeise
Rank
Userlevel 2
Badge +9

Couldn’t find anything in a web search for this exact error, so before I create a support ticket, I wanted to get forum moderator eyes on this.

Background: we take weekly synthetic fulls of our daily incremental jobs. Then at the beginning of each month, we look back 2 months ago, and age out any synthetic fulls that are not the last one of that that month. This way, we should have a single end-of-month synthetic full backup for each month, starting 2 months ago and going 12 months, after that point we age out and expire the oldest synth fulls.

Problem: a user asked me for a folder from 11 months ago. No problem, I think - that’s what our old synth fulls are for.

We’re using the Windows File Agent, so I “show backup history” on the subclient, scroll down to the bottom of the list, right click the synthetic full for June 26, 2021, and hit “browse and restore”

I immediately get an error in the browser: “No items found in the index, possibly index is being rebuilt. Please try later”

I then tested all my newer synthetic fulls. As it turns out, every synth full until 1/26/2022 gives me the same error. I.E. Jan 26’s synth full can be browsed without an issue. But the next-oldest one, December 25th, gives me this error and all other older ones do too.

As far as I can tell on my main mediaagent, the index is NOT being rebuilt.

This is very worrying. I trust commvault to hold onto my backups for 12 months, exactly as I specify. Why am I getting this error about an index, and what can I do to resolve this and get my user’s folder back?

 

Note: I have another subclient backing up a different drive on the same client. Exact same retention settings. These old synthetic fulls DO seem to be working (i.e. I can browse them). So it’s just this one subclient.

But I need the subclient in question working. I don’t like having to guess whether or not my backups are working.

icon

Best answer by ZachHeise 12 May 2022, 22:44

View original
If you have a question or comment, please create a topic

10 replies

Aplynx
Rank
Userlevel 6
Badge +13

The index is stored with the media of the backup. If you look at contents of a job on media you’ll see data and index. The index the browse references to work it’s way back is stored locally on the media agents and you can see the settings in the catalog tab. If the index no longer exists on the media agent it has to be restored to get to the data you are looking for. Due to a number of factors such as age, media type, amount of objects backed up, an index restore may take some time to complete. 

This should spawn an index restore job in the job controller. If there is not job spawned or the index restore job reports an error you should open a ticket for review. 

Liam
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Hi @ZachHeise , thanks for coming here first!

There’s a few possibilities, including that the index might be rebuilding already, but taking a while (I often see where the browse on older backups works later in the day upon retrying).

A few quick items to check:

  • Is there plenty of room for the restored Index?  I doubt this is the issue if others subclient browses work.
  • Can you try using another Media Agent?  
  • Can you see if any upgrade was done to the CS or MAs on or around this day?

There *IS* an undocumented registry key that MIGHT work, but it requires support to confirm first.  The key allows the recon to skip older cycles (which is what you can mention if you need to open a support case).

See if the above help, and if not, create a support case and share the number here for tracking 😎

https://www.linkedin.com/in/michael-struening
ZachHeise
Rank
Userlevel 2
Badge +9

Hi @Mike Struening and @Aplynx ,

Unfortunately we only have the one mediaagent (mediaagent01) for our oldest 12 month data. We have mediaagent02 in an offsite location but that’s only backing up the last 2 weeks in case of a disaster situation onsite; we’re okay with losing the full-year archive only in case of a disaster (had to cut costs somewhere after all).

Yes, the mediaagents have plenty of room on them.

Unfortunately in the Job Controller there’s no sign than any index rebuilds are being done either (i.e. nothing is being spawned).

How would I look at the “contents of a job on media” to see if there’s both data and index?

ZachHeise
Rank
Userlevel 2
Badge +9

We keep the index cache, mediaagent DDB, and jobresults folders as 3 separate folders on the root of a 3TB SSD in mediaagent01. Confirmed that there’s still 900GB of free space on that SSD.

ZachHeise
Rank
Userlevel 2
Badge +9

Argh! Okay I tried again to browse our oldest synth full and THIS time it worked in the Browse window. The restore also completed successfully.

Okay, so I guess I should have just been more patient. But it’s weird that it didn’t give me any sort of Job Controller feedback that some rebuild actually was occurring. Is there a log file I can make a note of to look for the status of a rebuild if I need to go back to my oldest indexes?

I did check my Catalog tab settings on MA01 and saw the setting for “index retention” only had “retain index for” set to only 15 days (probably again to save space) unless that’s unrelated.

I’m very relieved to have a successful restore done, I just wish things were less opaque about when commvault needs to do something in the background before an admin can browse.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Glad it's working!

Indexing logs are a bit complex (there’s more than one phase/log).  If it happens again, I would just right-click the job and view logs.  Any relevant logs will show up and allow you to confirm the restore is progressing.

https://www.linkedin.com/in/michael-struening
ZachHeise
Rank
Userlevel 2
Badge +9

Hi Mike - I would have done that, if a new job would have shown up telling me that an index restore was occurring.

That was my whole point; there was nothing that I could see in the Java console showing me that any sort of action was happening; all I could see was that I wasn’t being allowed to Browse certain synth full backups.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Ah, I’m sorry.  Yeah, so you essentially didn’t even get a job (for say, a restore) to even check.

If you need to do a quick check, start with these, though note you'll see the guid, not the subclient names: 

logmanager.log, indexstate.log, indexserver.log, and there’s also indexrestore.log

those cover the index, though you can also check browse.log on the CS to see what is happening at the top level.

https://www.linkedin.com/in/michael-struening
ZachHeise
Rank
Userlevel 2
Badge +9

Got it. I verified at 2:59 just now that I was having this issue still on a synthetic full job, same error.

On MediaAgent01 I opened up indexstate.log and saw that a new line was written:

21360 5cc0  05/12 14:59:05 ### DB [2EF7BB31-A4BD-4F02-A349-53455140199B] - Engine [1] - Status [IDX_DB_RECONSTRUCTION] - ReconJobcount [1]

I then waited until I saw the next line:

21360 275c  05/12 15:20:53 ### DB [2EF7BB31-A4BD-4F02-A349-53455140199B] - Engine [1] - Status [IDX_DB_SUCCESS]

And verified that now I can open the synth full job in question.

I will make a note of this for the future that if my support staff see this error, they should wait 20+ minutes and try again. Thanks.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Awesome, and glad to help!!

https://www.linkedin.com/in/michael-struening
Terms & ConditionsCookie settings