Hi All
We recently upgraded our commserve to 11.25 and since then the commcell / commserve server / backups are slow
The response time when accessing or doing anything on the commcell is dead slow.
Anyone facing this issue currently ? Any thoughts ?
Answer
Commserve Slow - upgraded to 11.25
- Novice
Best answer by Mike Struening
Looks like the case was archived with the following:
Jobs would intermittently fail to progress and build up in the job controller.
This issue would occur in random intervals since updating the environment to FR25
Many jobs would sit in a ‘Waiting’ state with no delay reason and no obvious reason (i.e. stream will be available & backup server will be reachable, but the jobs will be in waiting state for a very long time).
Additionally, during the time of the issue, some jobs would also ‘Fail to start’ entirely & logins to the Commcell Console would take very long to complete.
We applied recommended SQL Instance tuning for more optimal performance but the issues still occurred.
With development review, we identified that the CS was becoming overwhelmed with a client sided script which was being ran on SQL clients to check the status of backup history for those clients.
The script was running the below command from several clients:
qlist lastjob -c " & strComputerName & " -a Q_WINDOWS_FS -b defaultBackupSet -s fs_sqlbackup
Even though this script had been in place for several years without this level of negative impact, the CS DB had never been designed to tolerate hundreds of requests for query compilation in parallel.
To mitigate the issue:
The user account being used to run the script was disabled and the script was removed from the client points
The Commserve performance has remained stable since.
+22- Vaulter
Hey
I’m not aware of anything like that at all. I spoke with our SME in support who suggested opening a case as he is not aware either.
You know your environment so if you say the response is slower, then it is slower.
Can you share the case number here so I can track it?
https://www.linkedin.com/in/michael-struening
- Novice
There are other issues too since the upgrade:
- CPU high utilization spikes on the CS
- the description of the client is not visible at the console level. You can view it only on client computer → right click → view properties
- newly discovered snapshot VM’s are not displayed in the client computers list inspite of user settings being correct
+22- Vaulter
Very interesting….definitely curious as to the cause here…..could be an incomplete upgrade where a file(s) didn’t get updated properly, but I’ll defer to the engineer (looks like you are on the phone now)!
- Novice
Was also planning to reinstall the console or if possible repair the commserve package installation..
Sure will give it a try too
Today the commvault technician has enbled SQL traces and will be gathering logs based on tonight’s problematic backups
+12- Novice
If the cpu is high on the commserve, is if because of the SQL?
If that is the case, you can maybe try to run a new full dbmaintenance (on the other hand, the ugprade should also trigger that)
- Novice
When troubleshooting, full dbmaintenance was the first thing we did - it did not help fix the issue. And yes, SQL is also highly utilizing the CPU. Below are the 2 top takers since the upgrade was performed
EvMgrs.exe
SQL Server Windows NT - 64 Bit
- Novice
Adding on, since the upgrade we also started to see a sudden increase in "Unable to quiesce guest file system" warning errors
Too many issues at a time to take!
+22- Vaulter
- Answer
Looks like the case was archived with the following:
Jobs would intermittently fail to progress and build up in the job controller.
This issue would occur in random intervals since updating the environment to FR25
Many jobs would sit in a ‘Waiting’ state with no delay reason and no obvious reason (i.e. stream will be available & backup server will be reachable, but the jobs will be in waiting state for a very long time).
Additionally, during the time of the issue, some jobs would also ‘Fail to start’ entirely & logins to the Commcell Console would take very long to complete.
We applied recommended SQL Instance tuning for more optimal performance but the issues still occurred.
With development review, we identified that the CS was becoming overwhelmed with a client sided script which was being ran on SQL clients to check the status of backup history for those clients.
The script was running the below command from several clients:
qlist lastjob -c " & strComputerName & " -a Q_WINDOWS_FS -b defaultBackupSet -s fs_sqlbackup
Even though this script had been in place for several years without this level of negative impact, the CS DB had never been designed to tolerate hundreds of requests for query compilation in parallel.
To mitigate the issue:
The user account being used to run the script was disabled and the script was removed from the client points
The Commserve performance has remained stable since.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.