commvault.com commvault.com
Solved

Unusual job activity alerts after upgrading to 11.34.20

Ken_H
Rank
Userlevel 4
Badge +15

The evening of Friday April 6, I upgraded my CommVault installation from 11.32.38 to 11.34.20. Following the upgrade (which required reboots of almost all the client computers), the Friday evening backups seems to be running normally.  When I returned to work on Monday morning I have roughly 300 Alert emails regarding “unusual job activity” on various servers.  Weirdly, some of the alerts say zero files have been changed:

Anomaly Notification

An unusual job activity was detected in Job [751258], type [Root Size Change]. Details: Unusual number of file operations in client [INF-SQLP104] - created files: [0], deleted files: [0], modified files: [0], root size change of [0.00] GB

Does anyone know what the problem is and how to stop all this spam? Is there a way to turn off these notifications?

Thanks

Ken

icon

Best answer by Ken_H 18 April 2024, 16:09

View original

16 replies

V
Rank
Userlevel 2
Badge +5

Hi Ken,

 

 Sounds like there may be some issue with the alert in this instance. If you haven’t one already please open a support case for a more thorough investigation.

 

Kind regards,

Vance Sicherman

Commvault Support

Ken_H
Rank
Userlevel 4
Badge +15

Ticket 240408-789 created.

Ken_H
Rank
Userlevel 4
Badge +15

Update: This issue has been escalated to the development team.

Onno van den Berg
Rank
Userlevel 7
Badge +19

Hi Ken,

What a bad experience that you had following an upgrade to FR34 ,but thanks for sharing this with the community. Do you already know why you had to restart all clients following the upgrade? Was this after the upgrade of the CommServe backend or after the upgrade of the client agent? 

Onno

Ken_H
Rank
Userlevel 4
Badge +15

Hell Onno,

The upgrade problem that required reboots was centered around the client upgrades. According to the support analyst that handled ticket 240405-412 for this issue, the cause was “a .dll file being held open during the upgrade process” and the solution was to reboot the affected clients. The vast majority of clients (maybe 80%) needed to be rebooted.

Our plan for future Feature Set upgrades is to perform them during change outages just in case there’s a need for a reboot. To be honest, waiting until an outage window is best practice so I share the blame of having to reboot production servers during the week. Since we have a strategy to deal with the need for reboots during future Feature Set updates, my real concern with this post is the alert emails.

Ken

Onno van den Berg
Rank
Userlevel 7
Badge +19

Do you happen to recall which .DLL library file it was? Sounds like something related to a dotNET update or at least something related to the OS or was it an external process that caused the lock on the DLL?

Other recommendation I can give you is to first update supporting management systems or systems in a test environment before moving to production also on client level. We use this as a best practices and it has saved us from running into big problems several times already. 

Ken_H
Rank
Userlevel 4
Badge +15

The analyst I worked with never specified which DLL file was the problem.

Also: We don’t have a test environment for CommVault.

Ken

Onno van den Berg
Rank
Userlevel 7
Badge +19

Ok. That is a pity. I was also not pointing to a specific test environment for Commvault, but was pointing to clients who act as test clients for business application or supporting management systems like patch servers or jump boxes. E.g. something on where you can verify that the installation or update of the client side agent is executed without unexpected side-effects. 

Ken_H
Rank
Userlevel 4
Badge +15

Since the partially upgraded clients wouldn’t back up following the upgrade of the CommServer, I *assumed* that non-upgraded clients would have also failed to back up.  If that’s true then there’d be no advantage to upgrading non-prod clients first.  Perhaps I am mistaken.  I’ll keep your suggestion in mind the next time I do an upgrade.

Ken

Onno van den Berg
Rank
Userlevel 7
Badge +19

I don't understand, but maybe I'm interpreting it incorrectly. You upgraded the backend en started the upgrade on the first batch of clients which in the end failed to backup. You then already know the current state which is not ok, right? So, upgrading the remainders doesn't make sense then if you already know the first batch caused unwanted side-effects rights?

Ken_H
Rank
Userlevel 4
Badge +15

Good morning,

I upgraded the CommVault backend servers and then upgraded all clients simultaneously. Those clients that didn’t fully get upgraded then failed to process backups.  I *assume* that computers running the old client would also fail to back up.

Ken_H
Rank
Userlevel 4
Badge +15

Update: This problem has been resolved.  I am told by the support analyst handling ticket 240408-789 that this is a known issue with 11.34.20. The solution is to upgrade to 11.34.22.

I have put the 11.34.22 patch in and the problem has been resolved.

Ken

S
Rank
Badge +1

Can you only get .22 by starting a case or is it available for download?  I can’t seem to find it and I have the exact same issues you did.  

Ken_H
Rank
Userlevel 4
Badge +15

I used the java client and went into the root of the tree view on the left and did right-click > All Tasks > Add/Remove Software > Download Software. In the screen that appears, I selected Select Release and specified 11.34 Maintenance Release 22. There was no special permissions as far as I know.

Ken

 

S
Rank
Badge +1

Yeah, I have the job kicked off and running, it is just taking forever.  It has been two hours.  

Ken_H
Rank
Userlevel 4
Badge +15

My upgrade from 11.32 to 11.34 feature update took a long time and multiple clients needed reboots for the upgrade to complete.  The 11.34.20 to 11.34.22 maintenance update ran pretty quickly for me - maybe 20 minutes(?).

Ken

 

Reply


Terms & ConditionsCookie settings