DDB Maintenance Mode (Resync required/Resync in progress) explanation

  • 5 January 2021
  • 2 replies
  • 367 views

Userlevel 3
Badge +3

Uncommonly you may find your DDB partitions held in Maintenance Mode citing a need for a resync as per below:

 

Status: Maintenance Mode(Resync required)

 

There are also instances where the CS DB is aware the resync is taking place and as such will output the following status:

 

Status: Maintenance(Resync in progress)

 

This means there is currently a flag marked against the DDB within our Commserve DB that requires us to validate the DDB state given an inconsistency was detected and/or there was a Commserve DR (Disaster Recovery) taken place recently. A DDB in this state is not usable for backup operations and will not incur pruning of data. 

 

As per our documentation, a resynchronisation encompasses the following process:

 

  • The data in the DDB is validated against the CS DB to ensure that both the databases are synchronised

  • If the CS DB and the DDB are not synchronized, the resynchronisation process removes the additional data entries in the DDB to reconcile inconsistencies in the CS DB

  • When the DDB entries are removed successfully, the DDB is marked online and is usable once again

 

Source: https://documentation.commvault.com/commvault/v11/article?p=12562.htm

 

A common solution is to attempt a manual synchronization of the DDBs within the environment as outlined in the linked article above. In most cases this does not apply in your scenario as the option is greyed out as per below:

 

 

A resynchronization is triggered automatically and as such commencement can be seen via the Event Viewer section of the Commcell Console as per below:

 

Major    <EVENT_ID>       11/18/2019 3:38:50 PM    MediaManager    62:2534    Initiating resync on [<STORE_ID>] Deduplication Engines.

 

However, to help assist in confirming whether or not the resync is progressing successfully, we can review the MediaManager.log on the Commserve instance to make sure everything is hunky dory. In the below example, two DDBs (Store ID 1 and 2) on this particular MA have been marked for resynchronisation:

 

CS > \Commvault\ContentStore\Log Files\MediaManager.log

 

13692 d668  12/04 17:05:31 -----  SERVICE  [     ]  Store [1] picked for resync.

13692 d668  12/04 17:05:31 -----  SERVICE  [     ]  Store [2] picked for resync.

13692 d668  12/04 17:05:31 -----  SERVICE  [     ]  There are [2] stores waiting for auto-resync.

 

13692 19590 12/04 17:05:31 -----  SERVICE  [     ]  Number of stores to be resync-ed [2].

 

13692 19590 12/04 17:05:31 -----  SERVICE  [     ]  Resyncing store [1].

 

13692 19590 12/04 17:05:32 -----  SERVICE  [     ]  Resync pending for store [1].  SuccessSplitSummary [0]. Validation progress [0%]. Pruning progress [0%].

13692 19590 12/04 17:05:32 -----  SERVICE  [     ]  Resync status: Waiting [1] Pending [1] Succeeded [0] Failed [0] Unreachable [0].

 

13692 19590 12/04 17:05:32 -----  SERVICE  [     ]  Resyncing store [2].

 

13692 19590 12/04 17:05:33 -----  SERVICE  [     ]  Resync pending for store [2].  SuccessSplitSummary [0]. Validation progress [0%]. Pruning progress [0%].

13692 19590 12/04 17:05:33 -----  SERVICE  [     ]  Resync status: Waiting [0] Pending [2] Succeeded [0] Failed [0] Unreachable [0].

 

13692 19590 12/04 17:06:03 -----  SERVICE  [     ]  Resync succeeded for store [1]. SuccessSplitSummary [1]. Store will be made online.

 

13692 19590 12/04 17:06:03 -----  SERVICE  [     ]  Resync succeeded for store [2]. SuccessSplitSummary [1]. Store will be made online.

 

13692 19590 12/04 17:06:03 -----  SERVICE  [     ]  Resync store stats: Waiting [0] Pending [0] Succeeded [2] Failed [0] Unreachable [0].

 

We poll for a resync update every 30 seconds to confirm status of the operation running locally between the CS <> MA. Resync completion is also recognised by Event Viewer and. We'll be looking out for the below event:

 

Information    <EVENT_ID>         11/18/2019 3:39:21 PM    MediaManager    62:2503    Deduplication Engine [<DDB_NAME>] successfully re-synced and marked online.

 

By default, we will attempt to resync up to 5 times before declaring failure. Previously, network issues counted towards this threshold but in recent times this was excluded. Below is an example of a particular resync failure:

 

CS > \Commvault\ContentStore\Log Files\MediaManager.log

 

18500 4590  03/26 20:47:18 ---  SERVICE  [  ]  Store [1] not picked for resync. Reason [Deduplication database path unreachable].

 

Regardless, there may still be instances where you've since resolved a connectivity or access to the MA/DDB causing resync failure (Maximum resync attempts reached via Event Viewer in Commcell Console) and you wish to perform another attempt.

 

This is possible through Additional Setting [nMaxDDBResyncAttempts] which is to be applied against the CS (Link: https://documentation.commvault.com/commvault/v11/article?p=8681.htm) as per below:

 

Name: nMaxDDBResyncAttempts

Category: MediaManager

Type: INTEGER

Value: 30 (up to your discretion, any amount > 5 is recommended)

 

This will be confirmed also via our logging upon the subsequent resync attempt:

 

CS > \Commvault\ContentStore\Log Files\MediaManager.log

 

23528 5110  01/25 15:30:19 ------  SERVICE  [      ]  Reg key [nMaxDDBResyncAttempts] found. Setting maximum resync attempts to [<INPUTTED_VALUE>].

 

Without any further issues at this point, we should be able to successfully resync once again whereby the DDB will be flagged online. Don’t forget to refresh the Commcell Console (F5, refresh button and/or restarting the CC) for those using this interface as the entries may be stale if reviewing the DDB status.

 

If the issue does persist further please seek Commvault Support and raise a ticket for a deeper look. Please let me know if you have any further queries/concerns with the above information. Otherwise, if you found this guide helpful, don’t forget to upvote this post for visibility!


2 replies

Badge +2

what i can conclude from this is, there will be the data lose if CS DR restoration has been done? data written after DR time then all that data will be removed?

Userlevel 5
Badge +9

what i can conclude from this is, there will be the data lose if CS DR restoration has been done? data written after DR time then all that data will be removed?

 

Partially true.   The data could be recovered with media explorer but that would be a last resort.  

Longer story:  This is why DR backups are critical for recovery readiness.  I cant stress enough that customers should be using frequent DR backups or CommServe LiveSync to multiple hosts, as it better enables your ability to recover from a CS outage and leave as much of the environment intact.   When thinking about recovery readiness, most of the security and ransomware aspects come into play, but CS outage/loss must also be considered.     In my environment, i have CS livesync writing to an adjacent DR standby, and I also have one living in Azure.  

 

Reply