commvault.com commvault.com
Question

Restore Index v2 clients from tape

  • 5 December 2022
  • 20 replies
  • 1257 views
T
Badge +2

Hi,

I am currently testing a complete disaster recovery from tapes.

Usually, full backups are copied to tapes every week, so I took a tape set from 2 weeks ago as it was exported.

I restored the CommServe from Commvault Cloud.

New Media Agent and tape library were deployed and the tape set was inserted in the library.

Index v1 client recovery works fine.

Index v2 client recovery asks for tapes that are out of the current set to recall indexes, so I can’t restore anything.

I tried the catalog option but it takes days so it’s not viable.

 

Is there a way to ensure that a weekly tape set contains also the necessary index backups?

20 replies

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@Toine , can you restore by Job ID (i.e. just restore the whole job)?

Are you looking to restore data, or the entire server in one shot?

https://www.linkedin.com/in/michael-struening
T
Badge +2

Hi @Mike Struening ,

Restore by job is not an option as it only works for File System iDA, and most of my data are VMs.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Appreciate the detail.  That makes more sense.

We don't support live browse from Tape Media for v2 Indexing based VSA jobs (for Guest Files and Folders).

However, Full VM restores from tape will be fine. In order to accomplish this from a DR site, it will need to rebuild the V2 IndexDB for the specific VM being restored. This will require the Index Checkpoint to be written to that same DR Media (it is by default as it writes to the same Storage Policy) and all of its incremental Transaction Log File media tapes if they weren't rolled into the Index Checkpoint.

You might have better luck on the latest Feature Release but it could require identifying the specific issue the test DR situation is running into for your site.

https://www.linkedin.com/in/michael-struening
Damian Andre
Rank
Userlevel 7
Badge +23

Hey @Toine,

To expand on what Mike said - the architectures are very different for V1 and V2 indexing. With V1, (in general) the full index is located on the same media as the job. With V2 indexing the database for the index is kept in the index cache, where as only the action logs (incremental index) are stored with the job.  This massively reduces storage over time since you are not storing a full copy of the index cycle with every job as well as other benefits.

Periodically the action logs are applied to the main index database and captured during index backups.

So to get the complete picture with V2, you need the latest index database + possibly the action logs which pertain to the job you are restoring if they have not yet been applied to the index DB.

The full index database is backed up separately to the clients. The index backup is automatically sent to the same storage policy / plan as the client you are protecting to ensure they end up in the same physical location - but as stated, it those backups could end up on different media than the data you are trying to restore.

Unfortunately its a bit of a trade off - better performance and scale but highly likely you will need to also recall the media where the latest index backup was sent. Support has some options to be able to rebuild a partial index from the action logs included in each backup to support a restore but its more of an emergency situation solution. The best option will be to recall that tape that contains the latest indexing backup along with the tapes supporting the clients you need to restore. I’m going to see if we have a report or can suggest a cloud report to show last tape media bar code so you know which ones to recall in a DR situation before the CS is online.

 

T
Badge +2

Hi @Damian Andre ,

 

Thank you for your answer.

Maybe rebuilding a partial index would worth it, as this is an emergency situation (at least it is supposed to be). How would it be done? Cataloging all the tapes?

I would be interested in the report that could help identify the right tape.

For the moment, there 3 weekly tape sets that are rotated. I am pretty sure that tapes are forcingly erased when reinserted because of the index retentions which are too long.

Even if they were not erased, how to build the right tape rotation policy if we don’t know when full and depending action logs are written to tapes?

Do you think that complete DR from tapes is even possible? 

 

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@Toine for the report, leverage the Index Backup Retention Report (which is a report located on Cloud.commvault.com) to target the client/subclient you want to recover.  Tape Media throws in complexities to the DR recovery process for v2 Indexing for legacy clients working on BackupSet level IndexDBs. For clients leveraging Subclient level IndexDBs, we’ve optimized the recovery requirements so we can stage and create a limited view IndexDB compared to required the latest Checkpoint IndexDB for the BackupSet (and all transaction log files to bring it up to date).

Let me know if that helps!

https://www.linkedin.com/in/michael-struening
Damian Andre
Rank
Userlevel 7
Badge +23

Hey @Toine,

I spoke with one of the SMEs, and to further break down what Mike said. It appears that if you have subclient-level indexing for the client (which was a recent-ish addition) then partial index may be automatically done. The beauty of that is it can use the index stored with the job to allow a browse so it should not require any other media.

You can find out more about subclient level index (as opposed to backupset level) using this doc link: https://documentation.commvault.com/2022e/expert/117203_subclient_level_indexing.html - however it does not appear to support VSA job as yet which does not help your situation

Here is a screenshot of that report Mike mentioned where you can see which media was used for the the index backups so you can recall them. This is available from the reports on cloud.commvault.com so you would not have to have your CommServe online to find the info assuming you are allowing/sending health metrics to the CV cloud.

 

S
Rank
Userlevel 6
Badge +14

@Toine 

Adding an extra info here, for converting to Subclient indexing, make sure last 2 cycles for ALL subclients are not on tape.

When you run the Subclient Indexing Workflow, select “Custom Retention (cycles)” to 2 as below:

 

If you have already run the Subclient Indexing Workflow, in the Subclient Properties > Advanced for that specific Client, you would see the Index, there you can check and confirm the “Retain index in cache for” and number of cycles:

 

Best Regards,

Sebastien

T
Badge +2

Hi @Damian Andre , @Mike Struening  and @Sebastien Merluzzi,

 

Thank you for your answers.

I’ll have a look at the report, but as stated, I’m pretty sure that the tapes used to store the index are already rotated.

If I understand well, I’ll have to wait a new feature release before I can DR restore my VMs from tapes with the help of subclient index.

What is your advice in the meantime? Having a cloud copy of all Big Data Apps subclients?

S
Rank
Userlevel 6
Badge +14

@Toine ,

You can use Cloud or Storage LUN:

https://documentation.commvault.com/2022e/essential/43517_configuration_of_disaster_recovery_dr_backups.html

In our Documentation we have another option: 

https://documentation.commvault.com/2022e/essential/96317_full_system_recovery_virtual_server_agent_for_vmware.html 

Best Regards,

Sebastien

T
Badge +2

Sure, I know how to protect Commvault DR Backup or having a full VM DR solution.

In fact the last option is too expensive.

As we were quite sure that we could restore VMs from the tapes only and this is not the case, we are looking for cheapest way to let us be able to do it.

Damian Andre
Rank
Userlevel 7
Badge +23

Hi @Damian Andre , @Mike Struening  and @Sebastien Merluzzi,

 

 

I’ll have a look at the report, but as stated, I’m pretty sure that the tapes used to store the index are already rotated.

 

Hi @Toine - what do you mean by rotated? Commvault will absolutely not allow those index backups to be overwritten?

T
Badge +2

Hello,

Considering that the “Retain data until” date is passed, the team that handle tapes force erase the tapes, putting them in Scratch pool for the next week.

You are right that they are warned as they got this message:

 

YYZ
Rank
Userlevel 4
Badge +12

Hey @Toine,

To expand on what Mike said - the architectures are very different for V1 and V2 indexing. With V1, (in general) the full index is located on the same media as the job. With V2 indexing the database for the index is kept in the index cache, where as only the action logs (incremental index) are stored with the job.  This massively reduces storage over time since you are not storing a full copy of the index cycle with every job as well as other benefits.

Periodically the action logs are applied to the main index database and captured during index backups.

So to get the complete picture with V2, you need the latest index database + possibly the action logs which pertain to the job you are restoring if they have not yet been applied to the index DB.

The full index database is backed up separately to the clients. The index backup is automatically sent to the same storage policy / plan as the client you are protecting to ensure they end up in the same physical location - but as stated, it those backups could end up on different media than the data you are trying to restore.

Unfortunately its a bit of a trade off - better performance and scale but highly likely you will need to also recall the media where the latest index backup was sent. Support has some options to be able to rebuild a partial index from the action logs included in each backup to support a restore but its more of an emergency situation solution. The best option will be to recall that tape that contains the latest indexing backup along with the tapes supporting the clients you need to restore. I’m going to see if we have a report or can suggest a cloud report to show last tape media bar code so you know which ones to recall in a DR situation before the CS is online.

 

Hi Damian

Is there  a way to ‘force’ this behaviour, so the Main Index is ‘complete’ and standalone for an end of month tape

ie The ‘4 or 5’ tapes that capture a monthly backup are entirely standalone, and do not ask for weekly tapes or other tapes that may have been generated in the 1-4 weeks before the ‘Monthly Tapes’ are generated 

 

 

T
Badge +2

Hi,

 

I wanted you to know that we finally got this to work thanks to the Support.

The key is that you have to remove the index backups (Big Data App) from the tape auxiliary copies.

From that point, this kinda force Commvault to write indexes on tape, as if we were with V1 indexes.

When launching Browse and Restore, it runs an Index Restore job from the tape in the background and then we can browse data to restore normally. 

 

Thank you for your help.

YYZ
Rank
Userlevel 4
Badge +12

Hi,

 

I wanted you to know that we finally got this to work thanks to the Support.

The key is that you have to remove the index backups (Big Data App) from the tape auxiliary copies.

From that point, this kinda force Commvault to write indexes on tape, as if we were with V1 indexes.

When launching Browse and Restore, it runs an Index Restore job from the tape in the background and then we can browse data to restore normally. 

 

Thank you for your help.

Hi Toine,

 

Can I clarify this:

 

Are support saying 

  1.   remove the Index (by unticking the IndexServer association) in the Storage Policy Copy to Tape 
  2. BUT - Doing so- ‘Forces’ the Indexes to be written to Tape

 

Do you know whether the Index is ‘forced’ everytime the AUX Copy to Tape runs?

 

Like you - I’d like V2 capability, but want the Index on the same set of tapes as the ‘JOB’

 

 

 

 

 

T
Badge +2

Hi YYZ,

 

For 1 and 2 you understood well.

Yes, index is written to tape next to the data, as this is done with index V1.

 

If you don’t do that, Commvault will try to use incremental Index backup that was done on another tape. This is useless because we are most using selective copies with tapes.

 

Regards,

YYZ
Rank
Userlevel 4
Badge +12

Hi Toine

Thanks for the reply and confirming

Regards

R
Rank
Badge

Hi @Toine & @YYZ ,

did you remove the IndexServer association for tapes and did restore still work as you expected?

I spoke with Support a few weeks ago and they told me the following:

If Big Data Apps jobs are not copied. Then for browsing you may need to recall tapes that have action logs(previous backup jobs) to replay them.

 

Which is quite a different answer.

I still have the problem that tapes are not moving to scratch pool because one index backup blocks it. The index backup which doesn’t get aged contains data of a monthly copy (retention 12 months)  and resides on a tape for daily copies 🙄

Either I’m too stupid to understand index v2 altogether or index v2 is one of the worst decisions CV ever made. It is a real PITA (sorry to say that). With tapes its more a “helldex”.

Rgds

YYZ
Rank
Userlevel 4
Badge +12

Hi Ronald

I havent tested ‘unticking’ the ‘association’ for the Index Server when copying to Tape

(That is - Unticking then - forces a ‘Full index’ to be sent to Tape

Its still on my mind to test this to the level that I’m happy with 

 

 

 

Reply


Terms & ConditionsCookie settings