Solved

DDB increase in (CommServe Job Records to be Deleted)


Userlevel 4
Badge +15

I’m getting the following alert email roughly once an hour:

> Anomaly Notification

> The system detected an unusual drop in the pruning performance for the following databases in commcell <CommServer_Host>

> Deduplication Database     Reason

> HyperScale_Primary           inrease in (CommServe Job Records to be Deleted)

> CV Cloud Storage               increase in (CommServe Job Records to be Deleted)

> Please click here for more details.

When I follow the “click here” link, I see:

> 1   CommServe Job Records to be Deleted

This has been going on for a couple of weeks.  I don’t think an annoying email is a big enough deal to open a ticket for but I’d still like to clean this up.  Does anyone know what the problem is and how to fix it?  I’m sorry to say the CommVault help pages for this are not very useful. 

Ken

icon

Best answer by Jos Meijer 23 August 2022, 22:14

View original

14 replies

Userlevel 7
Badge +23

@Ken_H you likely have an increase in Pending Prune Records, or a lack of decrementing (so it’s stacking up higher).

You can try to cycle the MA services on the pruning Media Agents which should force action on the list.

You can monitor the deletions in the SIDBPhysicalDelete.log files.

Now, if nothing happens, either the MAs are busy with writing jobs, or something bigger is going on (and I’d create a support case).

Userlevel 4
Badge +15

My understanding is that my cloud services are connected to my HyperScale.  Last Wednesday I put the three nodes into service mode and rebooted them all then took them out of maintenance mode but the errors continued.  On Friday, my server admin patched and rebooted all the Windows servers within my CommVault installation… and the errors continued. 

Weirdly, after all that, the last error email was Sunday morning at 8:06 AM and I haven’t seen another since.  I’m not sure what fixed it as nothing seems to align with the time for the stop of the email alerts but since it’s no longer occurring I won’t be opening a support ticket.

Thanks for the help @Mike Struening, I really appreciate your quick response to forum posts.

Ken

Userlevel 7
Badge +16

When this happens there might simply be to much to data prune within the time frame until the media agent receives new prunable records.

This could be due to the batch size for which pruning occurs every hour (by default).

If the problem occurs again you could temporarily try the following additional setting on commcell level “MMpruneProcessIntervalMin” and set it to 30 (minutes). Advised is to disable it when done testing, this can cause performance impact on the media agent.

https://documentation.commvault.com/additionalsetting/details?name=%22MMpruneProcessIntervalMin%22&id=2135

 

Userlevel 4
Badge +15

Thanks @Jos Meijer,

About 3 weeks ago I _meant_ to change the configuration of my Tier 1 production backups to replicate NEWLY generated backups to Metallic cloud storage.  Instead, I mistakenly started replicating ALL backups to cloud storage - including monthly backups going a back full year.  The first copy job took about 9 days during which new daily backups continued to be generated.  The second job took about 2.5 days and eventually the aux copies caught up but somewhere along the line these alerts about “Increase in Job Records to be Deleted” started showing up.  As I wrote in the original post, I thought I’d given them a fair number of days to settle down but they didn’t give any indication the cause was going away - hence this thread on the forum.

Everything is good for me now so I don’t need to make any changes.  Nonetheless, I’ll mark your reply as the answer for the next person that happens to stumble across this.

Ken

Userlevel 7
Badge +23

Glad to hear it!

Userlevel 4
Badge +15

Thanks @Jos Meijer,

This happened again.  Starting last night at 10:30 PM I received a “increase in (CommServe Job Records to be Deleted)” alert email every hour for 12 hours.  I set the MMpruneProcessIntervalMin flag to “30” and within 3 hours everything was back to normal.  I’ve since deleted this additional flag.

Ken

Userlevel 7
Badge +16

Good to hear that this setting works to mitigate your situation 🙂
I would advice you though, if this occurs frequently, to evaluate the performance of the media agents and storage targets. Assuming a normal yearly data growth the frequency to mitigate with this setting will also grow.

Badge +1

I think i need to set the MMpruneProcessIntervalMin flag to 30 minutes to stop the same issue on my system. but after working with CommVault for almost 2 years, i feel like a total newbie.  Where do I go to enter that command or set that flag?  I have looked through what I can find in the java console and the web console, but don’t find much about flags.  Thanks in advance for your help!

jd

Userlevel 7
Badge +16

Within the left section of the Java GUI, right click on the commserve and open the properties. Enter the parameter on the addition settings tab 🙂

https://documentation.commvault.com/2022e/expert/8681_adding_additional_setting_from_commcell_console.html

 

Badge +1

@Jos Meijer Thanks so much for your help!  Looks like that took care of it.  

Userlevel 4
Badge +15

Just one follow up.  I kept having reoccurrences of the “Increase in CommServer Job Records to be Deleted” anomaly notification emails and decided that every time I got five of those alerts in a row, I’d decrease the MMPrunceProcessIntervalMin parameter by 2 minutes.  Starting with an initial setting of 60 (minutes), over the past two weeks I’ve slowly whittled that parameter down to 46.  I haven’t had an anomaly notification regarding this in the past 6 days so I think that’s the proper value for my installation.

Ken

Userlevel 7
Badge +16

Thanks for the follow-up @Ken_H 

Good to hear you could fine-tune the situation to get stable results 🙂🙂

Badge +1

I set mine at 30 minutes based on Jos’ post above and the notifications stopped for a day or two, so I thought I was in good shape, but then they started over.  I then dropped it to 20 min. but it didn’t fix it.  I’ll try Ken’s approach and see how it does.  Thanks!!

Userlevel 4
Badge +15

I originally set the prune interval to 30 minutes, the alert messages would stop, I’d set it back to 60 minutes to avoid the performance impact, and the alerts would resume in a few days.  After doing this three times I decided to slowly decrease the interval to try and find a balance between stopping the emails and avoiding (as much as possible) the negative effect on performance.

Ken

Reply