Skip to main content
Solved

DDB RECONSTRUCTION

Hello, please i have an issue with my DDB reconstruction. Not quite long when i move my DDB to another folder e.g folder1 to folder2 but on same server. 3 days later, my colleague wants to restart the server and force kill the SID process for process manager. and now the server went into DDB recovery. since one week now file system recovery will complete but adding records will fail . now today i found out the revoery process is taken from folder1.

reason becuase after DDB move the new path has not done DDB weekly backup.

now my question is according to commvault  https://documentation.commvault.com/11.24/expert/12582_moving_deduplication_database_to_another_location.html

now the file system recovery is pointing to folder1 instead of folder2 what do you suggest.

can i move DDB back to his previous folder1 ? and what could happen since it keep doing reconstruction and failing.

 

i log a ticket against support a lady came help still is still failing.

what other way can i move this DDB folder back to his previous path to enable recovery since recovery is pointing to old path which is becuase that was were all the previous DDB weekly backup has.

 

please all idea is welcome thanks.

15 replies

Jos Meijer
Rank
Userlevel 7
Badge +17

I don't expect moving the DDB back will work while the DDB is in recovery.

If the folder difference is the actual issue I would try to restore the DDB manually to the current folder2 and restart the media agent services. Then see if the reconstruction can complete.

An other way would be to get development to adjust the DDB folder configuration back to folder1

L
Rank
Userlevel 6
Badge +15

Hi and sorry for this bad situation.

This is not to blame anyone, but to remind (IMHO) best practices. DDB Backups, like index backups, and Commserve DRBackup, are to be performed often. 

In my environment, I do schedule them each 8 hours, so 3 times per day. It’s a bit time/IO consuming, but at least much useful if you have to recover your DDB after such operation or a crash.

I experienced multiple times some brutal SIDB / DDB processes stop/kill, and for once it took us more than a month to recover a huge partitioned DDB of almost a Petabyte of backend..

I had another issue , on a MA hosting multple DDBs, but only one was failed and had to be recovered, the global recovery process failed, because all DDBs were trying to be restored while one was OK, which was not a use case, and well it had to be solved ‘manually’ by the support team.

And another time, some guy just forgot this was an active MA processing backups, and just pushed the POWER (off) button of the physical server. Luckily after getting the MA back on, automatic recovery started, reconstruction took a few minutes as it occured during low activity.

 

This was just to illustrate that it can be easy or very hard to get back online when DDB has to be recovered/reconstructed. 

I’m convinced that the people working on your case @commvault support will manage your case the right way and assist to recover it in the most cautious way. If you share your support case ID, @Mike Struening may also help.

O
Rank
Userlevel 6
Badge +15

If the recovery will not work due to not having a current DDB backup, you can do a full DDB reconstruction.  This will not restore a DDB backup, instead it will rebuild the DDB from scratch.  It is a longer process but it will get you past the issue of not having a valid DDB backup.

A
Rank
Badge +7

Thank you guys. Well, I have also tried the full Reconstruction still failing.

O
Rank
Userlevel 6
Badge +15

Can you please post the error being seen?

H
Rank
Userlevel 2
Badge +5

@Alexzy 

The folder path being different will not cause the recovery to fail. You can always do a config only move to a different folder and the recon using the backup will still reconstruct it in the new path because its the folder structure under the folder you point to that matters and this will be correct in the backup.

For example:

Original Path was D:\DDB1. The folder structure under this would be D:\DDB1\CV_SIDB\2\X (where x is the store ID).

If you changed the path to E:\DDB2. When you reconstruct it will reconstruct the DDB under E:\DDB2\CV_SIDB\2\X.


Furthermore, the fact that the full reconstruction is also failing, shows that the issue is something else. If the failure is occurring in Add records, check job manager.log to see which MA is returning the error and check the DDBRecovery.log OR the ScalableDDBRecovery.log on that MA. This should show you the error.

Hope this helps.

Cheers

A
Rank
Badge +7

@Harsh Desai  Thank you so much for your rich advice.if i move D:\DDB1\CV_SIDB\2\X to

E:\DDB2\CV_SIDB\2\X. which reconstruction is still doing will it still work?

i also find out that path H:\ is having read error which i have granted read and write to the Admin.

 

Error occurred while processing chunk [1123876] in media [V_363047], at the time of error in library [BA_DISKDR] and mount path [[drmedia server1] H:\System Volume Information\DRMA01_LUN05], for storage policy [GDP_DR_BACKUP] copy [DR-DISK-COPY2] MediaAgent [10.100.81.57]: Backup job [46145]. Access to the file/media is denied.
Source: hq-vm-commserv, Process: AuxCopyMgr

i have check this path is accessing and please advice is there any way i get any script to ignore this partition.

 

Please advise.

 

 

 

A
Rank
Badge +7

What if I create a new DDB? What will happen to the old one? Will I be able to take my pending axillary job? Please  expert in the house. What did you think?

H
Rank
Userlevel 2
Badge +5

@Alexzy

In our current scenario, you moved the DDB from D:\DDB1 to E:\DDB2 and the DDB went corrupt before a good backup could be taken correct? If this is correct, when the reconstruction runs, it will reconstruct the DDB under E:\DDB2.

If you seal the DDB (which starts a new one) the aux copy will start running immediately HOWEVER you will not see any data pruning occur for any data tied to the sealed DDB as it was sealed in a corrupt state. Every last job tied to the sealed DDB will need to meet its retention before a single KB of data tied to the DDB is pruned. Restores will be unaffected as DDBs are not needed for restores.

P.S. if the reconstruction is failing I think that is something you may want to fix because sealing may not be an option every time the DDB goes corrupt.

A
Rank
Badge +7

What i did was  D:\DDB1 was move withing folder but same  D:\DDB1.

now my question is if i create new DDB what will happen to those job in old  D:\DDB1?

and can i still be able to do Aux copy?

A
Rank
Badge +7

What am thinking is to create another DDB what my worry is what will happen to the data in the old DDB?

 

H
Rank
Userlevel 2
Badge +5

@Alexzy.

Read operations are not affected by DDB being corrupted (unless the data is already corrupt on the disk library). If you seal the DDB and start a new one, you will still be able to restore data tied to the sealed/corrupt DDB but new jobs wont be able to reference the same blocks on the disk library so you may have up to double the data.

The only thing to keep in mind is going to be the baseline size for the existing DDB. As I said earlier, if you seal the DDB in a corrupt state, absolutely no pruning occurs for the data tied to the sealed DDB. For example lets say your current DDB has a baseline size of 50 TB and the library only has 40 TB of free space, if you seal the DDB, the library will run out of space as the original 50 TB worth of data is not going to be pruned until the very last job tied that DDB meets retention.

Below is a doc that talks about some of the use cases for sealing the DDB, the considerations and the overall process:

https://documentation.commvault.com/11.24/expert/12591_sealing_deduplication_database.html

I would like to stress that the reconstruction should work if there are no issues. I would recommend fixing the issue if possible because sealing may or may not be an option every time the DDB goes corrupt.

Regards

A
Rank
Badge +7

Thank you very much, @Harsh Desai  from the log I read shows, the DDB is corrupt. That's why we are having this issue.

Since Seal will not be the best option now, then what more can I try because I have run out of idea?

I plan to create another DDB, then move the DDB folder to it, and start.what is your advice on this.

A
Rank
Badge +7

Is it possible to run backup to Tape?

Not Auxillary copy i mean live backup to tape straight.

H
Rank
Userlevel 2
Badge +5

Alexzy

You can run a backup straight to tape however it would need to be using a storaage policy copy that does not have deduplication enabled as you cannot use the same policy copy with deduplication enabled to write to tape since it has deduplication enabled.

Regards

Reply


Terms & ConditionsCookie settings