I need your help and a few words explaining the problem that happened.
I have a schedule that executes a HealtCheck report. There wasn’t problem with it before, but it ended with a mistake twice. I analyzed the logs and there was only an error about connecting to the console. (sceen)
I would like to ask you to look at the problem and answer what could be the problem? Screenshots in the attachment.
Thank you for your help Regards Kamil
Best answer by Juan Rosas
Hi Kamil,
We will need to check what is happening in Webconsole.log at that time, reporting service is telling us that timeout was reached, but the cause of the timeout is not clear.
12812 3894 04/01 07:00:08 823631 genReportsWebSchedule::generateReport() - Generating report using web console [http://ilcs.xyz.pl:80/webconsole/]
12812 3894 04/01 07:18:08 823631 genReportsWebSchedule::downloadReportFile() - Download error message [0xEDD0001C:{CvCURL::CvProtocolReq::download(460)} + {CvCURL::CvProtocolReq::SendReqest(611)} + {CvCURL::CvProtocolReq::Execute(672)/CvCURL.28.0x1C-CURL perform failed.Curl ErrorCode : [28], Error Message : [Operation timed out after 1080000 milliseconds with 0 bytes received]}]
12812389404/0107:18:08823631 genReportsWebSchedule::generateReport() - Failed to generate report using any web console.
12812389404/0107:18:08823631 EvReportMgr::GenerateReportNonSecurity() - Failed to generate report for report type=[8012] task ID=[823631].
12812389404/0107:18:08823631 EvReportMgr::disposeReport() - HealthCheck : Report Failed
Curl ErrorCode : [28], Error Message : [Operation timed out after 1080000 milliseconds with 0 bytes received]
and then:
12812 3894 04/01 07:18:08 823631 genReportsWebSchedule::generateReport() - Failed to generate report using any web console.
12812 3894 04/01 07:18:08 823631 genReportBase::getReportUrl() - Generating report using web console [http://ilcs.xyz.pl:80/webconsole].
Is the Web Console server here the correct and expected server or has something changed in the environment? E.g. is the Commserve still able to communicate with the Web Console server on http:80?
We can use a setting to choose a particular Web Console for generating these scheduled reports if you need to:
Nothing was modified in the environment at the time the problem occurred, as you can see before and after the problem, HealtCheck Report generates correctly. The problem happened twice and I am looking for the cause of the problem.
We will need to check what is happening in Webconsole.log at that time, reporting service is telling us that timeout was reached, but the cause of the timeout is not clear.
12812 3894 04/01 07:00:08 823631 genReportsWebSchedule::generateReport() - Generating report using web console [http://ilcs.xyz.pl:80/webconsole/]
12812 3894 04/01 07:18:08 823631 genReportsWebSchedule::downloadReportFile() - Download error message [0xEDD0001C:{CvCURL::CvProtocolReq::download(460)} + {CvCURL::CvProtocolReq::SendReqest(611)} + {CvCURL::CvProtocolReq::Execute(672)/CvCURL.28.0x1C-CURL perform failed.Curl ErrorCode : [28], Error Message : [Operation timed out after 1080000 milliseconds with 0 bytes received]}]
12812389404/0107:18:08823631 genReportsWebSchedule::generateReport() - Failed to generate report using any web console.
12812389404/0107:18:08823631 EvReportMgr::GenerateReportNonSecurity() - Failed to generate report for report type=[8012] task ID=[823631].
12812389404/0107:18:08823631 EvReportMgr::disposeReport() - HealthCheck : Report Failed
at javax.servlet.jsp.jstl.core.Config.get(Config.java:126) at commvault.web.core.CVCoreUtil.getLocalizedStringForBundle(CVCoreUtil.java:378) at commvault.web.core.CVCoreUtil.getLocalizedStringFromDefaultBundle(CVCoreUtil.java:368) at commvault.web.core.Util.getLocalizedStringFromDefaultBundle(Util.java:213) at commvault.web.core.MainController.createServerErrorDetails(MainController.java:257) at commvault.web.core.MainController.processRequest(MainController.java:141) at commvault.web.core.MainController.doGet(MainController.java:296) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) 5488 170 04/01 07:00:42 192.168.128.70 1 INFO ScheduleFilter:?:? - ### invalidating JSessionId: SessionId: 8468922048DAC93AE8039AAE9FD57384 5488 156 04/01 07:45:15 192.168.128.70 4 ERROR ExportHandler:?:? - Failed to run PhantomJS:
5488 156 04/01 07:45:15 192.168.128.70 4 ERROR ExportHandler:?:? - Export failed. java.lang.RuntimeException: phantom: Process Timed out after 45 minutes
5488 156 04/01 07:45:15 192.168.128.70 4 ERROR MainController:?:? - MainController failed to process target runReportSchedule.do. IncidentId is: p4r1jk9jz2ghzu1bcxa. Exception is: java.lang.reflect.InvocationTargetException java.lang.RuntimeException: Error while exporting
5488 156 04/01 07:45:15 192.168.128.70 4 WARN ServerBroker:?:? - Invalid response returned: 401 5488 156 04/01 07:45:15 192.168.128.70 4 ERROR ScheduleFilter:?:? - JOBID: 823631. Error while invalidating the token and session after running schedule UNAUTHORIZED_REQUEST
We’re getting into the territory of needing more logs and probably a staged Commserve DB to investigate this further, so at this point, I recommend you raise a support case.
Please PM me the case number and I will track and hopefully update this thread so the Community may benefit from the resolution.
Sorry for the lack of updates on my part. The problem has stopped occurring at the moment, I will wait until Monday. If the problem starts to recur, it will create a ticket in Commvault support.
The problem has stopped occurring at the moment, perhaps one of the Hotfix versions has solved the problem. I left the case for observation, when something starts to happen I will let you know.
On June 3, an error occurred while generating the HealthCheck report. (screen 1-2). The problem was marked with error code 35:481 - the description is very general and does not indicate any particulars.
I have reviewed the logs initially, but I cannot determine what the problem may be. I am asking you for help in analyzing the problem.
@Kamil - from the logs that you had attacked - it is showing that there was a timeout when it was generating this report, - It looks like it had taken over 18 min before timing out.
8404 3a9c 06/03 07:00:02 873727 genReportsWebSchedule::generateReport() - Generating report using web console [http://ilcs.qwe.pl:80/webconsole/].
We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.