Solved

Commserve Slow - upgraded to 11.25

  • 11 October 2021
  • 9 replies
  • 574 views

Userlevel 2
Badge +2

Hi All

We recently upgraded our commserve to 11.25 and since then the commcell / commserve server / backups are slow

The response time when accessing or doing anything on the commcell is dead slow.

Anyone facing this issue currently ? Any thoughts ?

icon

Best answer by Mike Struening RETIRED 11 February 2022, 17:06

View original

9 replies

Userlevel 7
Badge +23

Hey @wstbackup !

I’m not aware of anything like that at all.  I spoke with our SME in support who suggested opening a case as he is not aware either.

You know your environment so if you say the response is slower, then it is slower.

Can you share the case number here so I can track it?

Userlevel 2
Badge +2

@Mike Struening  Thank you. Here is the case ID 211008-394

Userlevel 2
Badge +2

There are other issues too since the upgrade:

 

  • CPU high utilization spikes on the CS
  • the description of the client is not visible at the console level. You can view it only on client computer → right click → view properties
  • newly discovered snapshot VM’s are not displayed in the client computers list inspite of user settings being correct
Userlevel 7
Badge +23

Very interesting….definitely curious as to the cause here…..could be an incomplete upgrade where a file(s) didn’t get updated properly, but I’ll defer to the engineer (looks like you are on the phone now)!

Userlevel 2
Badge +2

Was also planning to reinstall the console or if possible repair the commserve package installation..
Sure will give it a try too

Today the commvault technician has enbled SQL traces and will be gathering logs based on tonight’s problematic backups

Userlevel 6
Badge +13

If the cpu is high on the commserve, is if because of the SQL?
If that is the case, you can maybe try to run a new full dbmaintenance (on the other hand, the ugprade should also trigger that)

Userlevel 2
Badge +2

When troubleshooting, full dbmaintenance was the first thing we did - it did not help fix the issue. And yes, SQL is also highly utilizing the CPU. Below are the 2 top takers since the upgrade was performed

 

EvMgrs.exe
SQL Server Windows NT - 64 Bit

Userlevel 2
Badge +2

Adding on, since the upgrade we also started to see a sudden increase in "Unable to quiesce guest file system" warning errors

Too many issues at a time to take!

Userlevel 7
Badge +23

Looks like the case was archived with the following:

Jobs would intermittently fail to progress and build up in the job controller.
This issue would occur in random intervals since updating the environment to FR25
Many jobs would sit in a ‘Waiting’ state with no delay reason and no obvious reason (i.e. stream will be available & backup server will be reachable, but the jobs will be in waiting state for a very long time).
Additionally, during the time of the issue, some jobs would also ‘Fail to start’ entirely & logins to the Commcell Console would take very long to complete.

We applied recommended SQL Instance tuning for more optimal performance but the issues still occurred.

With development review, we identified that the CS was becoming overwhelmed with a client sided script which was being ran on SQL clients to check the status of backup history for those clients.
The script was running the below command from several clients:
qlist lastjob -c " & strComputerName & " -a Q_WINDOWS_FS -b defaultBackupSet -s fs_sqlbackup

Even though this script had been in place for several years without this level of negative impact, the CS DB had never been designed to tolerate hundreds of requests for query compilation in parallel.

To mitigate the issue:
The user account being used to run the script was disabled and the script was removed from the client points
The Commserve performance has remained stable since.

Reply