Solved

New DDB stores gets Auto-created.. Horizontal scaling enabled..


Badge +4

The other day I noticed a Critical item in health dashboard under DDB backup section stating one of the store isn't protected.. Noticed the DDB store itself got auto created a day back keeping the old one active..  I noticed this across various environments where we have multiple DDB store with different ID gets created and all of them are actively used.. Couldn't find any documentation that explains it.. It would be helpful if someone throws some light here.. Thanks.. 

Below is the one i was referring to where DDB store 72 got auto created and if you notice for FS and DB agent store we have multiple ID’ present...

 

icon

Best answer by NVFD411 11 July 2022, 17:55

View original

12 replies

Userlevel 6
Badge +16

It sounds like the DDB has been sealed automatically and a new one is created.
The sealed DDB will be kept in use until all references to blocks are aged.
This is normal behaviour if the above scenario is the case.

Quote documentation:

 

“The active DDB can be sealed:

When the DDB is sealed, the sealing process closes the DDB and starts a new DDB. When the new database is started, the next instance of each data block processed creates a new signature tracking entry and the data block is written to the disk again as a new initial baseline.

The sealed DDB is automatically deleted only after all the jobs (with their data block signatures stored on the DDB) are aged or deleted and the corresponding volumes are deleted from the disk.”

 

I would check which setting is active for you, either the periodic or incidental scenario.

Badge +4

Thanks for the response Jos but as I said earlier these are not sealed.. This is online and actively being used as far observed.. And as verified when I view jobs for both the stores i see the same set of jobs present in both the stores..

Also if it is sealed then it would show me an Icon like below with sealed time, Right ?(picked these from an actual sealed DDB store) I don't see any of them in my case..

 

 

Userlevel 6
Badge +16

My apologies, I misinterpreted the information.

That is strange, never seen this behaviour before.
Seems that either:

  • The system failed to seal the old DDB, or
  • There are activities which should'nt happen, or
  • The GUI is showing incorrect data / the CS DB contains false information.

Do you see the ID's actively used in the SIDBEngine log?

As where the marked ID is the actual DDB Engine ID:

If so are all ID's used or only the highest number of each data type?

Badge +4

I see this behavior across many infra.. Below is another case where I see two DDB stores for VM’s..While verified sidb engine log the entries are from both the ID’s.. 

 

 

Userlevel 6
Badge +16

That's interesting 🤔

To be honest, I don't know what causes this. I have checked all my environments and all have one DDB ID per data type.

Curious if others know the reason, otherwise maybe development can answer?

Userlevel 1
Badge +5

@Prem Take a look at this setting in the control panel - DDB horizontal scaling threshold number of subclients per DDB 

If this has a value set, once it meets the requirements a new substore will be created -

https://documentation.commvault.com/11.24/expert/11020_media_management_configuration_deduplication.html

Userlevel 6
Badge +16

Good to know @NVFD411  😀

Thanks for letting us know 👍

Badge +4

Thanks for the information @NVFD411 , the value make sense but i checked across multiple infra’s and noticed the value is set to 0 even though could observe new DDB stores in place… Please refer below screenshot.. I am still curious how the new DDB store gets created, do we have any other condition as well apart from this that takes in effect….

 

Userlevel 4
Badge +11

Hi @Prem 

Could you tell us how many partitions are there and what is the Total Number of Unique Blocks on the DDB with lower ID?

Badge +4

In one of the environment there are 2 partitions, with below count of unique blocks.. And in all the other infra it is 3+ partitions..

 

 

 

Userlevel 4
Badge +11

I think one of the reason a new DDB store is created is when you have a high count of Unique Blocks, but these numbers are nothing out of ordinary, so I don’t think it’s the reason for new DDB Store.

You can check creation time of this additional store and if this is somehow fresh event then I would go and check event report if there any messages about stores created around that time. I would expect that something like that would leave a mark in event log and it may suggest a reason for the new DDB.

Badge

Default threshold would be for Q&I times above 1ms average for previous 30 days and number of primary table above 200 million records or primary table above 800 million records per partition.

 

The MM Config parameters mentioned are user defined, so these may be set in addition to the above thresholds.

 

 

Reply