Skip to main content
Solved

Commvault Azure Immutability

We want to have Azure Blob containers Immutable feature turned ON.

Primary - Incremental - 30Days - AzureBlobContainer1
Monthly Full backup Extended retention - 365Days - AzureBlobContainer1
With Commvault Deduplication.

Commvault DDB resides on the Media Agent.

  1. How so we handle DDB and the extended retention of INCR and Full on the Container1?
  2. Will the data on the container1 expire post INCR has reached its retention for 30Days?
  3. Currently we are not sealing our DDB. Do we need to seal the DDB when using Azure Immutable storage?
  4. Can we have one Azure Immutable blob and inside them have multiple folders with multiple retention?
  5. How does Commvault Retention linked with Azure blob Immutable - Access Policy - Retention Interval?
  6. What if Azure Retention Interval on AzureBlobcontainer1 is set for 30Days and My Commvault retention is for 365Days. what happens to the same data on 31st day?

11 replies

C
Rank
Userlevel 5
Badge +14

Hello @Kavya 

You can refer to our documentation for WORM on Cloud.

Configuring WORM Storage Mode on Cloud Storage - https://documentation.commvault.com/2022e/expert/9251_configuring_worm_storage_mode_on_cloud_storage.html

  1. Can you clarify what you mean by “handle the DDB”?
  2. The data will prune once the Storage Policy and Azure Lock duration have been met only when the DDB is sealed.
  3. Yes the DDB does need to be sealed at the same interval of the Policy retention. For example if the Policy has 60day retention, the DDB needs to be sealed every 60days.
  4. Azure Retention lock is container-based. You can only have 1 retention per container.
  5. The Azure Retention Lock duration must be set to twice the amount of time as the Policy. For example, the Policy has 60day retention, the DDB is sealed every 60 days, the Retention Lock must be set to 120 days.
  6. The data won’t age until after 365 days and the DDB has been sealed, but after 30 days it can technically be modified.

Note:

1) All of this is automated by the workflow and all you need to do is create the Library\Storage Pool, Create the Copy and associate the storage pool, run the workflow on the storage pool.

2) All Copies associated to the storage pool must have the same retention configured to prevent issues with pruning.

 

Thank you,

Collin

K
Rank
Userlevel 2
Badge +4

Why and how is the DDB sealing linked to immutable backups

K
Rank
Userlevel 2
Badge +4

Thank You @Collin Harper, That was very helpful and in-depth reply

I have a follow up scenario

Primary Copy

AzureBlobContainer1 with Immutable
Primary Copy 30 days incremental with dedupe
Primary Copy Full backup at the end of month with dedupe

The Azure Retention Interval is 30 days
Commvault retention is 30 days

How will the DDB sealing work?

Secondary Copy 

Aux copy the full backup from AzureBlobContainer1 (primary) with dedupe to a AzureBlobContainer2 with dedupe with immutable.

The Azure Retention Interval is 180 days
Commvault retention is 180 days


How will the DDB sealing work?

M
Rank
Userlevel 2
Badge +6

Hello @Kavya ,

I also discussed a lot with Commvault and tested in the last year regarding Immutable Copy in Azure.

At the moment we test concrete following Requirement:

  • 7 Day Immutable Copy → All Jobs
  • Using Object Locking mechanism

In Azure you have the following Requirements for Using Immutable with Object Lock:

  • dedicated Storage Account with
    • Versioning Enabled (all Versions) → no cleanup Rule by Azure
  • Hot or Cold
    • If you have less than 30Days Retention you should us Hot, because on “Cold” you have “Minimum Stay Days” from Azure of 30 Days
    • So we use “Hot”
  • Create a Container with “Enable version-level immutability support”

Important to know about WORM/Immtuable :

  • for each Retention you must have an own Bucket → not mix Retetions in the same bucket
  • Snapshot Copy Jobs not supported (Job based Retention)
  • Micro Pruning is not supported
  • Immutable Object Lock always calculated Retention *2 , this means if you have 14 Days Retention the Immutable Lock is 14*2 = 28 Days
  • DDB Sealing necessary because of Cleaning Date must use Macro Pruning
    • Background is that all referenced blocks in the DDB must be locked up to the final release and thus the aging can take place only in the whole with sealed DDB.

After you added the Immutable Container as CLoud Storage you must run the WOrkflow ”Enable WORM Storage” This WF sets:

  • Disable Micro Pruning
  • Set WORM LOCK Days 2* Retention
  • Sealing DDB to Retetion Days (in Default - can be overwritten)
  • Set WORM Active on Storage Policy associated with Library → no Deletion possible

 

In our Case we added to our Storage Policy a “Worm Copy”  “All JObs” with Retention “7Days / 0 Cycles”

  • Retention Days set correct
  • Object Lock works
  • DDB get sealed as configured
  • Deletion of Data in Container not possible

At the moment we Monitor the Costs in Azure and we have one problem open Index Jobs are not pruned and therefore Sealed DDBs not cleaned up (I asked today here in the forum for help)

 

The most difficult thing is cost calculation, but maybe you can get in contact we Commvault from your side with Manager, they have a Excel Cost Calculator to estimate Cost, but this Calculator so far as I know is not public.

Hope this helps a little bit.

BR

Marcel

 

C
Rank
Userlevel 5
Badge +14

@Kavya 

The Azure Retention Lock would need to be double the duration of the Commvault retention. This is because of deduplication. 

Example: A Job is written on Day1 would age on Day 30. A Job written on Day 30 would age on day 60. Since everything is deduplicated the job written on day 30 depends on the blocks written by the jobs written days 1-29. If retention lock is set to 30days and the DDB is sealed, all the blocks from Days 1-29 become eligible to prune. You would be pruning the data from the job before its retention is over and it wouldn’t be restorable.

DDB sealing is required because we cannot micro-prune the data. Micro-pruning requires the files to be modified as we delete the unnecessary data within the blocks. WORM prevents the files from being modified so we can only prune once all of the data has aged. Sealing the DDB allows it to be prunable.

 

Thank you,

Coilin

M
Rank
Userlevel 2
Badge +6

@Kavya short Update here: we had one problem open with Index Backup Jobs got bot pruned, this was a BUG:

we were running on V11.28 then the fix is included with V11.28.76

 

indexing backups are not aging from sealed DDB

 5065, 5066

 

After installing this fix release, Index jobs got pruned and sealed DDBs were cleaned up.

But have to keep in mind Index Jobs always have different Retention settings.

BR

Marcel

Nikos.Kyrm
Rank
Userlevel 3
Badge +13

Hello @Collin Harper  and @Marcel Geisen,

For 365 backup (infinity retention), with backup destination a hot Blob Storage, could you please advise me about the immutability / WORM copy procedure?

Is just the WORM copy checkbox in Storage policy or are there additional steps needed?
 


Please for your feedback,
Nikos

C
Rank
Userlevel 5
Badge +14

Hello @Nikos.Kyrm 

For SP32 and below, you need to run the workflow.

Configuring WORM Storage Mode on Cloud Storage - https://documentation.commvault.com/2022e/expert/9251_configuring_worm_storage_mode_on_cloud_storage.html

 

For SP32 and up, it is configured in Command Center through several settings.

Enabling WORM Storage and Retention for Cloud Storage - https://documentation.commvault.com/2023e/essential/155900_enabling_worm_storage_and_retention_for_cloud_storage.html

 

Thank you,
Collin

Nikos.Kyrm
Rank
Userlevel 3
Badge +13

Hello @Collin Harper

Thanks for your reply!

I see for SP32 and below, in WORM Storage provided documentation, o365 workloads are not supported!

Also for SP32 and up, Worm Storage still not supported for o365 workloads, I double check it with Commvault Support team (231127-230).

So, WORM Storage currently the Exchange Online Agent, OneDrive for Business, SharePoint Online and Microsoft Teams agents are not supported for WORM. 😔
 

I thing maybe the WORM Copy should be the way to proceed it with o365.
Just check the check box in Storage Policy level.
https://documentation.commvault.com/2022e/expert/13938_worm_copies.html


Best regards,
Nikos

C
Rank
Userlevel 5
Badge +14

@Nikos.Kyrm 

The following Agent’s are not supported as of SP32.

Configuring WORM Storage Mode on Cloud Storage - https://documentation.commvault.com/2023e/expert/9251_configuring_worm_storage_mode_on_cloud_storage.html

The following agents are not supported for WORM:

  • Dynamics 365

  • Azure Active Directory

  • Exchange Online Agent

  • OneDrive for Business

  • SharePoint Online

  • Laptop IDA

  • Microsoft Teams

  • Exchange Mailbox Agent

  • File System Archiver Agent (Windows, Unix, NAS)

  • Any Object Store (Edge Drive, third-party migration, and so on) and hybrid file

  • Snap copy (job-based retention)

 

Thank you,
Collin

A
Rank
Badge +7

Hello @Kavya ,

I also discussed a lot with Commvault and tested in the last year regarding Immutable Copy in Azure.

At the moment we test concrete following Requirement:

  • 7 Day Immutable Copy → All Jobs
  • Using Object Locking mechanism

In Azure you have the following Requirements for Using Immutable with Object Lock:

  • dedicated Storage Account with
    • Versioning Enabled (all Versions) → no cleanup Rule by Azure
  • Hot or Cold
    • If you have less than 30Days Retention you should us Hot, because on “Cold” you have “Minimum Stay Days” from Azure of 30 Days
    • So we use “Hot”
  • Create a Container with “Enable version-level immutability support”

Important to know about WORM/Immtuable :

  • for each Retention you must have an own Bucket → not mix Retetions in the same bucket
  • Snapshot Copy Jobs not supported (Job based Retention)
  • Micro Pruning is not supported
  • Immutable Object Lock always calculated Retention *2 , this means if you have 14 Days Retention the Immutable Lock is 14*2 = 28 Days
  • DDB Sealing necessary because of Cleaning Date must use Macro Pruning
    • Background is that all referenced blocks in the DDB must be locked up to the final release and thus the aging can take place only in the whole with sealed DDB.

After you added the Immutable Container as CLoud Storage you must run the WOrkflow ”Enable WORM Storage” This WF sets:

  • Disable Micro Pruning
  • Set WORM LOCK Days 2* Retention
  • Sealing DDB to Retetion Days (in Default - can be overwritten)
  • Set WORM Active on Storage Policy associated with Library → no Deletion possible

 

In our Case we added to our Storage Policy a “Worm Copy”  “All JObs” with Retention “7Days / 0 Cycles”

  • Retention Days set correct
  • Object Lock works
  • DDB get sealed as configured
  • Deletion of Data in Container not possible

At the moment we Monitor the Costs in Azure and we have one problem open Index Jobs are not pruned and therefore Sealed DDBs not cleaned up (I asked today here in the forum for help)

 

The most difficult thing is cost calculation, but maybe you can get in contact we Commvault from your side with Manager, they have a Excel Cost Calculator to estimate Cost, but this Calculator so far as I know is not public.

Hope this helps a little bit.

BR

Marcel

 

Hello,

I see there a contradiction: 
“Immutable Object Lock always calculated Retention *2” and "If you have less than 30Days Retention you should us Hot, because on “Cold” you have “Minimum Stay Days” from Azure of 30 Days”

If we need retention of 15 days and more, Immutable Object Lock will be 30 days or more. For me it sounds that you should use cool instead of hot starting from 15 days retention. Or I'm missing something?

Reply


Terms & ConditionsCookie settings