Solved

Problem with data pruning after seal DDB


Userlevel 4
Badge +11
  • Commvault Certified Expert
  • 83 replies

Hello Guys,

 

I need yor help.

The Customer has sealed a DDB.

However, they ignored the following warning:

There was not enough free space on DL.

As a result Disk Library is out space of now and the Customer has disabled schedules to not run any backup.

As a troubleshooting step we tried to delete some jobs assigned to sealed DDB.

AS you can see, there are over 20,000 jobs assigned to this DDB.

We deleted over 12,000 jobs.

Hovewer, Commvault is still showing the same number of jobs and no data has been pruned from DL.

Is any way to force Commvault to free some disk space removing data assigned to sealed DDB?

Rgds,

Kamil

icon

Best answer by Kamil W 14 May 2021, 09:04

View original

13 replies

Userlevel 4
Badge +9

Hi @Kamil W,


Have you run data aging after marking those jobs to be deleted?

  • Some scenarios will prevent micro pruning of the data in a DDBand will not reclaim any free space until all jobs are pruned.
  • If you have not run data aging this is the reason that the data continues to linger after marking the jobs.
  • If you run data aging but no further space is reclaimed then you may need to wait until all data is pruned from the DDB.
  • You can aux copy any jobs that are still necessary to another location if needed for a longer duration
  • Then select all remaining jobs if you are certain you no longer need them and mark them to be deleted
  • Once data aging is run it will mark all jobs deleted from the database and the physical pruning will begin
  • Once all pruning is completed for the sealed DDB this will remove itself from the Commcell console completely
Userlevel 4
Badge +11

Hi Sean,

 

Thanks for your reply.

 

Yes, I have run data again, even few times.

 

“ Some scenarios will prevent micro pruning of the data in a DDB and will not reclaim any free space until all jobs are pruned. “ - this is exactly what I expected when the Customer called me.

 

Anyway, I will get back to the Customer with your answer and I’ll let you know how it goes ASAP.

 

Rgds,

Kamil

Userlevel 4
Badge +11

Hi @Sean Crifasi,

 

The Customer has deleted all jobs assigned to sealed DDB.

Data aging has been run but no data has been removed from DL.

I can see that Ttoal Application Size is now 20 MB (previously was 384 TB).

However, Total Data Size on Disk is still the same (134,69 TB)

Now:

 

Previously:

What else the Customer can do to remove the data from Disk Library?

 

Rgds,

Kamil

Userlevel 7
Badge +23

@Kamil W , can you check and share the SIDBPhysicalDeletes.log files on the Media Agent(s) connected to this library?

Want to see if it is actively deleting or if there are any errors.

Userlevel 4
Badge +9

The data size on the sealed DDB may not necessarily progressively change as it prunes. Additionally the DDB tabs do not dynamically update and you may not see a change until you close and reopen the Commcell Console or refresh with F5 on the DDB Engine level.  

As Mike suggested, please check SIDBPhysicalDeletes and possibly SIDBPrune log as well for the Media Agent in question for the DDB partition. for a total size on disk of over 100tb this could take several hours or more to completely prune all data on the sealed store depending on the performance of the Media Agent.
 

Userlevel 4
Badge +11

Hi Guys,

 

Thanks for the update.

 

SIDBPhysicalDeletes.log and SIDBPrune.log habe been uploaded.

 

I am just reviewing them as well.

 

Rgds,

Kamil

Userlevel 4
Badge +11

I cannot see any errors.

 

I have just open a ticket with support. I’ll let you know guys, how it goes.

 

Rgds,

Kamil

Userlevel 7
Badge +23

Keep us posted.  The log file shows a LOT of activity, though an engineer on a remote can see a lot more detail.

I found the case and will track its progress.

Userlevel 4
Badge +11

Hi Guys,

Please find the below information that I got after the remote session with the support.

Investigation:
-----------------
 

  • Since the DDB reconstructions got failed, the previous DDB was Sealed. After which the jobs are not pruned from the Sealed DDB and space reclaim didn't happen.
  • Verified that no jobs are listed from the Sealed DDB.
  • Noticed that there are no pending prunable records on the Sealed DDB as below
  • Noticed there are 2 jobs not pruned from archjobOnStoreInfo table  
JobID- 1010466 and 1012254
 
  • Ran ‘qoperation agedata -delbystoreid -sid 11’ followed by data aging once and confirmed that the jobs are pruned successfully.
  • Suggested to wait for some time to reclaim the space on the library.

 

Userlevel 4
Badge +11

I spoke to the Customer yesterday. It looks like pruning is going fine now.

 

Rgds,

Kamil

Userlevel 7
Badge +23

Thanks for confirming!!!

Badge

What does error “Failed to prune CCR” mean?? Is there any resolution steps?

Userlevel 7
Badge +23

@vishakha that indicates a record pruning error.

Are you receiving this error?  We should spawn a separate thread to track your issue better.  Can you confirm and provide some additional details?  Where are you seeing this and when?

Reply