DDB Priming with Auxilary Copies

Hello ​@Robert Horowski 

You are correct that there is very little on documentation around this feature.

The wording of this feature makes me think that it is only for backup jobs and not Aux copy operations. 
I can see Damien's post does reference Aux copy jobs but i have not seen it used for that practice and there is a difference between Dash copy ( aux copy with Dedup on both sides ) and a source side backup job. 

I would recommend documenting all your findings and raising a support ticket. they can get an official response from Dev in regards to this functionality and its limitation. From that ticket a Doc MR will be raised to get Books online reflecting the correct details so people in the future do not hit the same wall as you.  

Kind regards

Albert Williams

Hello ​@Albert Williams 

Appreciate your post!

On high level it seems like both operations - backup with source side dedup and DASH copy - have a lot in common, but I agree that this may be misleading and in the end only devs know how does it work under the hood.

It is a shame though, because you could really hit two birds with one stone and even if these processes are more different then what it seems, one could think that a simple workaround should be possible i.e. after Sealing DDB for WORM lock just AUX last Full to newly created DDB. All of the pieces are there but there is no process to handle this.

Anyway, since this is a design phase I cannot really open a support case, cause there is no support contract in place yet. It looks like I will just have to figure out how to do things differently.

Hello ​@Robert Horowski 

I have sent an unofficial email to the Dev team asking the question and if there is a limitation. 
No guaranty they don't have a rule for me yet putting my emails to the Trash but i will keep you posted!

It is also Xmas time so I am sure it wont be a quick response. 

Kind regards
Albert Williams

Hello ​@Albert Williams ,

Thanks, it is much appreciated.

Fingers crossed for your name on devs white list in 2025!

Please, let me know once they will get back to you.

Have a happy new year!

Kind regards,

Robert

Hello ​@Robert Horowski 

Can you confirm if you have WORM enabled at all? There is a new feature with WORM that once the DDB is sealed it gets purged. If this were to happen there would be no references available for the priming when the new job runs. 

Kind regards
Albert Williams

Hello ​@Albert Williams 

Yes, the WORM is enabled on secondary copy

There is a new feature with WORM that once the DDB is sealed it gets purged. If this were to happen there would be no references available for the priming when the new job runs. 

If that is what is happening, then it makes sense, that jobs succeeding DDB sealing need to be fully copied to secondary copy. Unfortunately this new feature defeats the ability to have a central, WORM locked Vault, without the need to periodically re-send Full over the long distance wire.

Is there an Additional Settings that could disable this new feature maybe?

I've run some additional tests:

1. AUX copy to secondary copy with WORM lock disabled.
   
   the sequence here is as follows:
   Full backup on client, AUX Copy, Seal secondary DDB, another Full backup on client, AUX Copy.
   
   This should allow to prime against sealed DDB but this is not happening.

   

   We still don't have an answear if DDB Priming is supported with AUX copy so this may be the reason.
    
2. Full backup to a primary copy with WORM lock enabled.
   
   If this new feature kicks in, I would expect that a Full backup after sealing DDB, would need to re-send data over the wire, just like in original scenario with AUX copy to WORM enabled secondary copy. This, however is not happening. After sealing DDB with WORM lock enabled on primary storage, next Full is not sent over the wire, only the metadata from what I can tell.

So either I got this wrong or it does not work as expected.

Currently in my lab I am on 11.36.35

Hello ​@Robert Horowski 

Thanks for all the testing and prompt responses. 

I have gone back on my thread that i have with Dev and really hammered the question ( Caps lock on ) “Do secondary copies support DDB Priming”. This is the root of our question and needs to be answered before we look into other things. 

I will keep you posted!

Kind regards
Albert Williams

Hello ​@Albert Williams 

Of course, since I am the interested party it cannot be otherwise! But also, I really do appreciate your time and effort to help me, so keeping you waiting is the last thing I wanna do.

Let me know once you hear from dev or if there is anything else I need to check or test.

Thanks,

Robert Horowski

hi Robert 

Sorry for the late arrival to the thread - but to answer the basic question - yes, any dedupe store can employ the priming option.   As this was an older feature combination it has not made the transition to the Command Center easy config yet.  As you note if you set this up on the DRC store ( assuming this is a WORM object or file lock library type), you can configure the priming option on the destination store.  

With the rise of WORM storage pools to secure backups against RW risks - we have been enlisting some of the older features that are now operationally relevant again.   The priming with WORM is a good example.

It is very effective in delivering the outcome you are looking for.  The WORM storage option when configured with the DRC store will periodically seal the ddb collections into vaults and orient all the data to be locked base on the basic retention days + ddb period.  It forces the macro pruning of the collection, which normally eliminated the need for the old DDB / that is where the seal and sweep option was used.  As Albert pointed out WORM was setting the sweep of the old sealed DDB to recycle that DDB volume space.  But in the primary case you need it to exist for a while longer to help reference links to the data in the prior vault (sealed DDB) so they can be copied forward into the new DDB; that is the role of the priming option.  As DASH copy runs (especially when it hits the first batch of Full or SF jobs into the new DDB/vault) it will trigger the destination MA at the DRC to reach back and copy the required segments from the prior store, avoiding the transmission from the robo source copies. 

This is on the list for migration to Command Center but for the moment you can use the quick option below to set it up on the destination store.

To help illustrate.

You will need to set the options in the DDB advanced settings for the destination DRC dedupe store.

This older BOL link offers a some context on setting the options in command line for the  DDB advanced settings

https://documentation.commvault.com/commvault/v11_sp20/article?p=14152_1.htm

1. Enable (set to 1) the DDB Priming option on the Storage Pool (DDB settings: useDDBPrimingOption )       

• Use this parameter if you want to refer to previously sealed DDBs to identify and use existing data blocks to prime the baseline for the current DDB.
• To enable DDB priming:   qoperation execute -af <download location>\Update_DDB_Settings.xml -copyName xxx -storagePolicyName xxx -useDDBPrimingOption 1                  
• • -storagePolicyName "<StoragePool Name>"
  • -copyName "<Storage Pool copy name>"
  • Names provided in the Data Retention (SP/Copy) report.                      

2. Disable (set to 0) the sealed Archive DDB rule (DDB settings: enableSIDBArchive )         

• To disable delete the DDB once a DDB is sealed:   operation execute -af <download location>\Update_DDB_Settings.xml -copy name xxx -storagePolicyName xxx -enableSIDBArchive 0
  • storagePolicyName "<StoragePool Name>"
  • -copyName "<Storage Pool copy name>"
  • Names provided in the Data Retention (SP/Copy) report.          

3. Confirmation of the settings and check if the Sealed DDB is present.

We’ve been enlisting this for service again in many new WORM storage deployments.

Cheers

brock

 Hi ​@Brock 

Late or not, I am very glad you joined us!

I really appreciate the detailed description of the feature, this makes it easier to understand and follow the whole process.

Not wasting any time I tried to run this in my lab. To do so on my existing MediaAgents I created new pools, setup a plan with secondary location, applied settings you mentioned in your post and run new backups. Unfortunately it seems like I still can’t get it to work as expected and I still can see data flying through the wire whenever I seal target DDB.

The test was as follow:

1. Run Full backup on client (JID 371)
2. Run Auxilary copy to SPC: Pool006_DR(WORM ON) (JID 372)
3. Seal Deduplication Database - Pool006_DR-ma2_Files_46 [ID: 46]
4. Run Full backup on client (JID 373)
5. Run Auxilary copy to SPC: Pool006_DR(WORM ON) (JID 374)

You can tell that DDB was sealed by looking at “DDB store” column.

You also can tell that data was transferred looking at “Transferred” column.

Looking at the cv-ma1 and cv-ma2 in real time shows that data is indeed transferred through ethernet interface

More about the setup:

Primary copy is part of a Pool006-ma1 created on cv-ma1 server, WORM lock is disabled on this Pool

Pool006_DR(WORM ON) is a secondary copy and it is part of a Pool006_DR-ma2 created on cv-ma2 server, WORM lock is enabled on this Pool.

I’ve been sealing DDBs like crazy but data is still flying between servers.

----------------------------------------------------------------------------------------------------------

Not sure if relevant, but I wanted to comment on the thing you mentioned:

As you note if you set this up on the DRC store ( assuming this is a WORM object or file lock library type), you can configure the priming option on the destination store.  

I do not have any supported WORM lock device at hand, both, source and destination Pools, are created on a regular vmdks (my mediaAgents are virtualized), with WORM enabled on destination Pool, but not on source. I may be wrong here, but - in the context of DDB Priming Feature - since it’s just a filesystem instead of a protocol (like in S3 cloud library case) I don’t think it should matter whether underlying storage is actually WORM capable here or not.

Hope I have not missed anything.

Is there anything else I could check?

Regards,

Robert

Hi ​@Brock ​@Albert Williams 

Is there anything else that comes to mind I could check to make this work? Let me know.

Let’s set up a live call - it will be easier to sort it out and then we can reply back to close the thread and share the results.  I’ll connect on the email option to set up a time - cheers b

Hi ​@Robert Horowski ​@Brock 

Did you ever get to the bottom of this?  I’m very curious to hear the outcome!

Regards,
David

Hi ​@David Chadwick 

Not really, Brock was busy at the time and we lost touch. I still haven’t figured out how to make it work as expected.

Regards,

Robert