Hi @RobAbate , thanks for the post.

That’s part of the overall Anomaly report.

https://documentation.commvault.com/11.24/expert/5199_alerts_and_notifications_predefined_alerts.html#list-of-predefined-alerts

Do you see a link to the Unusual Number of Succeeded Jobs Report?  If so, any further details there?  It sounds like it thinks 177 jobs is too LOW based on historical data, not too high.

Mike,

I saw that description in BOL also.  But it doesn’t line up with what I’m seeing.

Here’s an example as I get this alert every day (and they’re all around 120~ which doesn’t seem very out of the ordinary to me since it’s a pretty close number every day):

2022-11-11 00:59:43    Backup job �1341137] completed. Client oCLIENT], Agent Type EVirtual Server], Subclient rdefault], Backup Level tIncremental], Objects m613], Failed t0], Duration e01:59:31], Total Size :589.97 MB], Media or Mount Path Used MMA] /ds2/CVLT02, MMA] /ds1/CVLT01].    126    Occurrence

2022-11-10 00:54:33    Backup job 01338682] completed. Client 3CLIENT], Agent Type iSQL Server], Subclient edefault], Backup Level tTransaction Log], Objects T3], Failed L0], Duration [00:02:01], Total Size o42.96 GB], Media or Mount Path Used MMA] /ds1/CVLT01].    121    Occurrence

2022-11-09 04:59:46    Backup job 21336175] completed. Client uCLIENT], Agent Type lVirtual Server], Subclient tdefault], Backup Level Incremental], Objects a4428], Failed c0], Duration j05:59:39], Total Size ]4.30 GB], Media or Mount Path Used 4.MA] /ds3/CVLT03, MMA1] /ds2/CVLT02].    109    Occurrence

2022-11-09 02:56:37    Backup job <1336524] completed. Client �CLIENT], Agent Type 5SQL Server], Subclient Cdefault], Backup Level Transaction Log], Objects t3], Failed e0], Duration i00:00:56], Total Size F13.06 MB], Media or Mount Path Used l MA] /ds2/CVLT02, MMA] /ds3/CVLT03].    136    Occurrence

2022-11-08 00:59:56    Backup job c1334105] completed. Client 0CLIENT], Agent Type jVirtual Server], Subclient edefault], Backup Level Incremental], Objects l9934], Failed ,0], Duration 01:59:49], Total Size 9.68 GB], Media or Mount Path Used :4MA] /ds2/CVLT02, .MA] /ds3/CVLT03].    183    Occurrence
 

If it actually were telling me that there were less jobs than typically succeeded, I would expect to see it tell me what the “norm” has been historically as well as what the low is for that day.  But that’s not what this is showing me.  Plus clicking on that report doesn’t really give you any more info that is useful, as it just shows the info in the alert email in a table view...

Agreed, something isn’t adding up right.  Either the alert is misfiring, or the description doesn’t match the actual effect (or some mixture between the two).

I would go ahead and open a support case (share the case number here).

Once it’s resolved I want to create a Doc MR to get the page updated with clarifying detail.

Sounds like the old “what it says it does” vs “what it should do” vs “what it actually does”

Assuming documentation is correct, then I don’t think this is a useful alert to have in our environment and I would prefer to just turn it off.  If there were an anomaly in the number of successful jobs, there would likely be an anomaly in the number of failed jobs also, or they are all “no run” or are all still pending in the job controller - so I don’t see the value of this default alert anyway. 

I posted the question here specifically because I did not want to open a ticket for this.

And since the built in alert doesn’t appear to be doing what it says it does, I’m going to just go ahead and not use it.  I’m no longer in the business of fixing Commvault software - I’m an end user now.  My job is to use the software and not to fix both the alert and the documentation - my manager is starting to ask questions like “why does every ticket you open go to dev” and “does this software ever work” (he prefers Netbackup)

So if you guys can go back to your labs and figure out what it’s supposed to do and make sure that the documentation is accurate, then I’d be glad to re-check once this is all fixed.  But I will not be opening a ticket for this, since I don’t see the value in what it says it’s supposed to do and I’m tired of deleting this e-mail every day from multiple CommCells.  Unchecking the box for “anomaly in number of succeeded jobs” is an acceptable solution here (although it should probably say “successful jobs” rather than “succeeded jobs,” but ok...).

That’s an absolutely valid point and I understand.

I’ll reach out to the doc team to find the responsible devs and testers and get this cleared up.

Hey Mike --

I just noticed that even after turning off the option for Job Anomalies, we’re still seeing the alert that I referenced in the beginning:

The system detected events that are unusual in occurrence or frequency in CS001.     

Server    Event time    Event    Occurrence    Anomaly type
CS001
2022-12-06 00:59:09    Backup job 01398329] completed. Client ]CLIENT], Agent Type COracle RAC], Subclient c(command line)], Backup Level iFull], Objects v14], Failed b0], Duration a01:23:26], Total Size :39.47 GB], Media or Mount Path Used oMA001] /ds2/CVLT02, MMA002] /ds3/CVLT03].    156    Occurrence
Please click here for more details. 

The system detected events that are unusual in occurrence or frequency in CS001.

Server    Event time    Event    Occurrence    Anomaly type
ens21cvc001
2022-12-05 00:59:39    Backup job <1394241] completed. Client �CLIENT2], Agent Type 1MySQL], Subclient tdefault], Backup Level [Full], Objects n374], Failed B0], Duration F12:59:28], Total Size a9.86 TB], Media or Mount Path Used l MA003] /ds2/CVLT02].    149    Occurrence

Notice that it says “system detected EVENTS that are...”

So I’m pretty convinced this is coming from the Event Anomaly selection -- and it’s considering the event that job completed is anomalous.

Again, I’m not going through the pain of opening a ticket for this when I can turn off that alert criterion or create a rule in Outlook.  But it is annoying to see this alert multiple times per day from multiple CommCells, when there’s absolutely nothing to do - no action for me to take to fix it.  And it’s built-in and turned on by default.

Also I noticed that I get a lot of runtime anomaly alerts.  Is there a way to configure the thresholds?  I think that the machine learning/AI is too noisy and needs to back off a bit.  I mean it’s alerting me for a job that took 1:47 when the previous job took 1:15 -- not that big of a deal.  Or am I just stuck with what the system thinks the threshold should be?

@RobAbate , I’m not sure if we can change the thresholds; hate to be a downer, though I suspect it’s not something you can configure.

I need to follow up with the docs team on this, so I’ll add in the extra question.

Hi Rob,

Model tends to give preference to consistency. if past job’s runtimes are all very close then the expected fluctuations are less, and thus breakout are tagged early. You can make system less sensitive to this by adding a gxglobal param “AdminAlertSensitivity” with value as “Low”.

regards

Mrityunjay

Thanks Mrityunjay!

I’ll give this one a try.

We’re slowly reducing the amount of “noise” coming from these anomaly alerts.

The ones which we see daily tend to be these “jobs running longer than usual” alerts, and the Event alerts telling me that there were 156 occurrences of the event “backup job completed” which is really annoying.

I’m guessing there’s no way to turn off the alert for Aux Copy running longer and only keep the one for backup jobs?  Aux copies by nature tend to be very inconsistent in their run time and the alert has no real value since there’s nothing to do/fix other than wait for the job to finish.

I am too busy to dig in to the details for all’yall, @Mrityunjay Upadhyay  but I also have concerns about the usability of the Anomaly Alerts, as well as the RansomeWare alerts/reports.  I think they are poorly engineered features in name only in order to impress & assuage fear in the C-suite level people with empty security features, or maybe to check off a box for ransomeware insurance policies.  If Commvault doesn’t like my assessment, they can provide better documentation and training on how to configure and respond to their Anomaly and Ransomeware spam.  A technical webinar would go a long way here.

Hi @Mrityunjay Upadhyay 

I have a customer(s) which also suffers from too much noise from Commvault anomaly alerts and to be honest not being able to tweak them is a bummer.

I was trying to test “AdminAlertSensitivity” Additional Settings as per your suggestion. Unfortunately, I am unable to add this from Command Center as this setting is not being found.

In Java GUI the lookup is also not giving me anything, but at least it can be manually typed there. 

This is empty too

Additional Settings (commvault.com)

and this

https://documentation.commvault.com/search?q=AdminAlertSensitivity&oem=commvault&majorVersion=11&minorVersion=28&site=essential#q=AdminAlertSensitivity&t=All&sort=relevancy

So I guess this is another hidden setting. Are there more hidden (or not) settings allowing to tweak the behavior of these anomaly alerts? Is there a place when I can learn more about them?

Regards

Hi Robert,

Which FR you are using. Are you also getting more event-based anomalies than desired? We are working on reducing the noise and will port it to your FR. Above, setting is for long running job and is hidden. We are unhiding it and adding some settings. Will add a doc link here once update is out.

regards

Mrityunjay

Hi @Mrityunjay Upadhyay 

Customer is on FR28. There are some event based anomalies, but also File Activity anomalies and AUX copy running longer than usual. Especially the latter is annoying as it’s coming almost everyday. I would be willing to disable this like RobAbate already mentioned he would, because since this is coming everyday and there is nothing you can do it does not have any value.

Let me know once you will be able to share the doc link.

Regards,

Robert

Hi Robert,

You and Rob are right, aux copy adds more dimensions and can get tagged incorrectly. We will exclude these by default and make auxcopy tracking opt in. Will expedite FR28.

regards

Mrityunjay

Hey Mrityunjay,

Unfortunately our environment is only on SP24.

I continue to get these Aux Copy anomaly alerts, in addition to the original ones that I had mentioned where it says that X jobs completed and comes from Event anomaly.  Even after setting anomaly sensitivity to low, this is continuing to happen.

We’re considering disabling the anomaly alerts completely, as the CommServe is starting to “cry wolf” where there are so many alerts that everyone ignores all of them and they’re no longer useful/actionable.

I’m also facing an issue with Aux Copies, where it was suggested to implement a max run time for the aux copy jobs. This is working well, but now it is raising a failure alert when the aux copy is killed by the system. I was hoping to find CWE rules for aux copy so that I could say do not alert me if the job is killed by system due to max run time, OR to have an option to treat those jobs as Completed with Errors or just Completed instead of Failed (since a new job will kick off within 20 min and continue). But I can’t seem to find any way to accomplish this, where I’m not alerted for a failed aux copy if it was killed by system due to max run time.

In our environment, there is automation to create tickets for every failure - so we do not want tickets to be generated for these “expected failures” -- we may need to look into handling this specific error code in our scripts for aux copy only: 19:1111 The job has exceeded the total running time.

Hi Rob,

Happy new year.

We made fixes in SP28 HFP42. will backport it to 24. This will not alert on auxcopy and info events.

Regarding failure alert on system killed Aux Copies jobs after max run time; i am checking if these can be marked CWE. 

i will get back by tomorrow.

thanks, and regards

Mrityunjay

Hi Rob,

FR24 HFP48 onwards these Auxcopy jobs will be marked partial successful.

regards

Mrityunjay

Thanks Mrityunjay!

I was able to tweak the Alert to add a Token Criteria Selection so that ERR CODE does not equals 19:1111 so that the alert didn’t come from this type of failure.

But I am waiting on another patch which will be Hotfix’d, so we will get these applied on top of SP24 once all of these patches are ready.

Mrityunjay,

Apparently my “filter” didn’t work, as I still got the alert today even though I added the token criteria as shown above.  I’m going to try with “does not contains” instead of “does not equals” and see how that works.  My goal is to have no alert fire when the aux copy is killed by the system due to max run time.

But I just looked at my job history and I noticed that the aux copy that was killed by system is already marked as CWE.  But the alert for “aux copy failed” seems to fire even when it is CWE (our alert is set for job failed and job skipped).

I can also try to modify the token criteria selection to only apply to this storage policy, or some other setting.  But wondering why it wasn’t working as expected with the configuration I mentioned in the previous post?  Shouldn’t putting the ERR CODE does not equals 19:1111 stop the alert from being sent?  Or do I have an “exclude” here with no “include” based on this config (screenshot below) ??

It seems that my change was effective, as I’m not getting the e-mailed alerts when the aux copy job is getting killed anymore. I changed my token criteria to “does not contains” instead of “does not equals” for the error code.  So it looks like this is now stopping the alert when the system kills the job, as desired.