Solved

WHY IS OUR MEDIA SERVERS ALWAYS DOING Dedup DB Reconstruction

  • 26 November 2021
  • 3 replies
  • 416 views

Badge +7

we just fix one of our media server that went into Dedup DB Reconstruction and now another one 

this always happen after Disaster Recovery Backup what is happen ?

icon

Best answer by jgeorges 27 November 2021, 00:23

View original

3 replies

Userlevel 5
Badge +9

Hey @Alexzy 

To confirm, when you say DR Backup, we’re referring to the Commserve DB Backup? If so, this is definitely unusual. BUT if we’re referring to the DDB Backup, than this could be something related to the VSS Process which may require a little more digging.

Either way, definitely worth raising a support case, however here are some fundamentals that might help identify the root cause:

  • SIDB2.exe process on the Media Agents is our DDB Process.
  • We use a bit marker, to identify when we’ve opened and closed a table. So:
    • If any of the many thousand DDB tables are closed ungracefully, the DDB is marked corrupt.
    • If any of those DDB Tables are opened by any process other than SIDB2.exe than they are marked corrupt.

I’d start by checking the SIDBEngine log around the time of the last shut down/reboot.

Find the current Process ID (its the first from the left OR can be found in Process Manager), scroll to the TOP of the log file and search down for that PID.
We don’t actually want to look at the current PID, but this gets us to the last moments before previous PID went down, and should let us see if the process was gracefully stopped before the power down/service cycle OR if it closed out abrubtly.

There are a few caveats to the log review above;

  • You need to confirm the DDB’s Store ID from the GUI, as every DDB on the Media Agent will write to the same SIDBEngine.log.
    • The Store ID can be found in the DDB details under ‘Storage Resources > Deduplication Engines > <DDB Policy> > <Deplication Database>
    • From the log you’ll see the Store ID after the Date/Time and JobID(or #####) appear something similar to 1-0-1-0 (EngineID, Group, Substore, Split)
  • If you find the last PID went down gracefully, then confirm it started in a good state as it may have already been corrupted:
    • Can search for the word ‘corrupt’ 
    • Can look for a line similar to:
      •  DDB State is [3]. Time [1521093739], Last PID [9420]
        • If the state is anything but [0], its broken.
        • Time is EPOCH time.
        • Last PID was the last good PID

If you do log the ticket out, let me know here and we’ll take a look for you.

Cheers,
Jase

Badge

Hey @Alexzy 

To confirm, when you say DR Backup, we’re referring to the Commserve DB Backup? If so, this is definitely unusual. BUT if we’re referring to the DDB Backup, than this could be something related to the VSS Process which may require a little more digging.

Either way, definitely worth raising a support case, however here are some fundamentals that might help identify the root cause:

  • SIDB2.exe process on the Media Agents is our DDB Process.
  • We use a bit marker, to identify when we’ve opened and closed a table. So:
    • If any of the many thousand DDB tables are closed ungracefully, the DDB is marked corrupt.
    • If any of those DDB Tables are opened by any process other than SIDB2.exe than they are marked corrupt.

I’d start by checking the SIDBEngine log around the time of the last shut down/reboot.

Find the current Process ID (its the first from the left OR can be found in Process Manager), scroll to the TOP of the log file and search down for that PID.
We don’t actually want to look at the current PID, but this gets us to the last moments before previous PID went down, and should let us see if the process was gracefully stopped before the power down/service cycle OR if it closed out abrubtly.

There are a few caveats to the log review above;

  • You need to confirm the DDB’s Store ID from the GUI, as every DDB on the Media Agent will write to the same SIDBEngine.log.
    • The Store ID can be found in the DDB details under ‘Storage Resources > Deduplication Engines > <DDB Policy> > <Deplication Database>
    • From the log you’ll see the Store ID after the Date/Time and JobID(or #####) appear something similar to 1-0-1-0 (EngineID, Group, Substore, Split)
  • If you find the last PID went down gracefully, then confirm it started in a good state as it may have already been corrupted:
    • Can search for the word ‘corrupt’ 
    • Can look for a line similar to:
      •  DDB State is [3]. Time [1521093739], Last PID [9420]
        • If the state is anything but [0], its broken.
        • Time is EPOCH time.
        • Last PID was the last good PID

If you do log the ticket out, let me know here and we’ll take a look for you.

Cheers,
Jase

 

Hi @jgeorges,

I have one query that does Commserve DB contain information of deduplication databases of Media agent?

Thanks.

 

Userlevel 3
Badge +6

Hello @shashikant,

 

The CommServe indeed tracks some details of every deduplication store and if those differ between the CommServe DB and MediaAgent, the DDB is pushed into reconstruction.

 

This also happend when you do a CommServe Recovery Tool action or unplanned failover using livesync and other scenarios as outlined by @jgeorges .

 

Regards,

Mike

Reply