Deduplication on Independent Copies for Azure Cloud Backups

Badge +5

Hello guys. I’m looking for some advice/tips on how best to configure additional selective copies in a storage policy and ensure they are deduplicated to avoid rewriting the same blocks on cloud storage. The Primary Copy is deduped and goes to Library 1. I want Weekly and Monthly copies to go on Library 2 and 3 respectively with each copy disabled. I noticed I can’t use the Global Deduplication Policy being used by the Primary Copy on the additional copies. Anyone has some thoughts on how to tackle this? I’m not a fan of using Extended Retention on the Primary Copy as and set Weekly and Monthly retention on one media/point of failure. 


Best answer by Mike Struening RETIRED 5 May 2021, 16:14

View original

6 replies

Userlevel 7
Badge +23

Hey @Baba Imani !  I have a few questions:

  1. Would these Aux Copies go to the same library?  Is library 2 and 3 the same cloud library (but logically different) or different locations altogether?
  2. What is your end goal for retention?

Note that a Global Deduplication setup needs to use the same library:

Creating a Global Deduplication Policy

The global deduplication policy enables deduplication across multiple storage policies that use a common set of disk libraries and deduplication database (DDB).

We generally advise against Extended Retention on Deduplication.  You’ll end up increasing the size of the DDB.  However, this is likely your only option.

We don’t allow multiple copies in the same Storage Policy to use the same Global DDB….it would be a single point of failure for both copies.

Take a look at the docs in the links above and let me know if they help or if you have more questions.


Badge +5

Thanks for your response. Well the libraries are different in the sense that each Library is Blob storage in its own storage account. So no the aux copies won’t be going to the same library per Ada each goes to its own storage account with Local Redundant Storage. Why I’m also against extended retention is because I want to syncro copy a deduplicated Primary Copy to another region and if I have Ex Retention I’ll end up copying across data for all copies. 

The end goal for retention is

Dailies - 15 days to Primary Copy On Library 1 (deduped by Global DDB) Archive tier

Weeklies - 35 days in Library 2 (deduplicated) Cool/Archive tier

Monthlies - 13 months on Library 3 (deduplicated) Archive tier

*What are the ramifications enabling  deduplication on the secondary copies using a different folder location on a dedicated disk that isn’t the existing Global DDB disk when I create the policy copy? 

Userlevel 7
Badge +23

@Baba Imani , thanks again for the extra detail!

With your goal, you won’t be able to Dedupe across the same Global DDB, which is fine.

You can absolutely set up different DDBs for each copy with your desired results.  

Are you hosting the Media Agents locally, or in the cloud as well?  If all MAs are in the cloud along with the DDB and library locations, you should be in a great place.  You don’t want to put yourself in a place where you have to pull data out (egress charges) and then send it back up.

Badge +5

Thanks @Mike Struening. Yeah we have two MAs in the active cloud region and 1 in the DR region. I will go with setting up separate DDBs for each copy and last question. I was thinking to perhaps not deduplicate the Yearly copy as it only happens once a year. As for Access tiers and deduplication what’s the best practice given my intended setup. Say for the monthly copy with 13months retention and deduplicated. Was thinking Cool/Archive.

Userlevel 7
Badge +23

I would agree with Cool/Archive.  I imagine neither of them will be accessed all that often so these should suffice. 

Regarding deduplication, if you are only keeping it for 13 months, then you’d only be deduping against the previous backup (ignoring common blocks across clients).  It will definitely cut down on space, though not at some large percentage.

Userlevel 4
Badge +9

@Baba Imani Are we okay to close this off as an ongoing discussion within “Creating a Selective Copy Using a Library (Enable Deduplication)?