commvault.com commvault.com
Solved

ddb backups missing files during backup

  • 14 February 2024
  • 6 replies
  • 111 views
roc_tor
Rank
Userlevel 1
Badge +6

So, I’ve got a ticket in for this, but have only recieved remedial solutions.  Basically, when we back up the dedupe database, its snapshot appears to be happening ok, but it’s missing files, as in they’re simply not there (see below).  

 

It’ll take days of downtime (hundreds of servers not backed up) if we simply need to rebuild this.  

Could this be a corrupt DDB?  Anyone else seen this issue?

 

 ERROR CODE [6:43]: Backup failed: unable to open or read a file. Configured to fail on any error. Please check the following: 1. The account that the File System agent uses has sufficient privileges to back up files. 2. The data being backed up can be accessed and is not on a path that is unmounted or is inaccessible. 3. A third-party product is not locking the files. 4. There is no corruption of the data on the disk. 5. If some failures are expected, please change the current setting so that the backups can complete.
Source: MyMediaAgent1, Process: clBackup

Files failed to back up:

·  [D:\ddbfolder\CV_SIDB\2\1\Split00\23042\Secondary_11797804.idx] [0x2 2] The system cannot find the file specified.

·  [D:\ddbfolder\CV_SIDB\2\1\Split00\23284\Secondary_11921644.idx] [0x2 2] The system cannot find the file specified.

·  [D:\ddbfolder\CV_SIDB\2\1\Split00\23048\Secondary_11800825.idx] [0x2 2] The system cannot find the file specified.

·  [D:\ddbfolder\CV_SIDB\2\1\Split00\23217\Secondary_11887393.idx] [0x2 2] The system cannot find the file specified.

·  [D:\ddbfolder\CV_SIDB\2\1\Split00\23284\Secondar...

 
 

 

Best wishes!

 

 

icon

Best answer by roc_tor 27 March 2024, 14:50

View original

6 replies

Hyder
Rank
Userlevel 3
Badge +8

Anything in the OS Event logs at this time for VSS?

Zubair Ahmed Sharief
Rank
Userlevel 3
Badge +9

Antivirus Exclusions are they in place ?

If you have a DDB backup before the issue occurred, we can attempt to recover the DDB and try backups again. 

Other things to verify, is it happening for only one MA ? Is it happening for one client backups only ? Any other pattern that can narrow down the issue or help with any clues for resolution?  Check with your security team for any weirdness with AV or ransomware
 

“Never settle for anything less than your best.” — Brian Tracy
C
Rank
Userlevel 5
Badge +14

Hello @roc_tor 

This error is not from the DDB partition being corrupt, but it occurs when there was an issue with the VSS Snapshot. Can you check Event Viewer for VSS errors? Also can you confirm if AV exclusions are in place? It’s also possible that AV is preventing us from reading from the snapshot.

https://documentation.commvault.com/2023e/essential/antivirus_exclusions_for_windows.html

 

Thank you,

Collin

roc_tor
Rank
Userlevel 1
Badge +6

Servers - we have two servers, this is only happening on one.

Event Logs - VSS Service.  Nothing out of the ordinary.  Just lots of VSS service is shutting down due to idle timeout.

Zubair:  Nothing with virus (and we do have the exceptions set up).  It’s Sentinal One (S1)… As for other patterns, this is just a snapshot of the ddb drive and it’s commvault files that are “missing”.  While we do ahve FS agents on various things, most of our setup is VMs and nothing like THIS is happening.  

 

The way I see it - our backups are really useless on the dayS (sometimes it works, most of the time it doesn’t) it doesn’t work.  

 

I just picture the shadow not taking.. and the FS agent is backing things up a live FS.  That’s what it LOOKS like, but the shadow’s taking.

 

 

roc_tor
Rank
Userlevel 1
Badge +6

Collin & Zubair - I’ve seen the AV exclusions, but everything still just smells like AV.  I’ll try turning it OFF.  Let’s see what happens.

roc_tor
Rank
Userlevel 1
Badge +6

Ok.. I think I have the answer for this one.

This drive is FULL.. and it is busy.  Something like 2.7 TB and only has a few hundred gig free (yes.. I know...).  The snapshots would supposedly take, but often they couldn’t because of writer problems.  When shadows did “work”, we had the symptoms I’m complaining about.

I moved the shadowstorage to another drive.  Problem went away.  

 

So this appears to be my answer:  When you’re overloading a drive, move the shadowstorage somewhere else less busy.  Bonus.. I get the space back for the DDB.

Reply


Terms & ConditionsCookie settings