commvault.com commvault.com
Solved

Generation of reports tend to fail. 11.24.29

  • 4 January 2022
  • 17 replies
  • 1191 views
L
Rank
Userlevel 6
Badge +15

Hi and best wishes to you !

 

This may not be related to it at all, or not, I do not know, but as most of us, we upgraded in late december our versions, and on my side I updated up to v11.24.29. I was at v11.24.23 before the log4j panic :wink:

 

Then I went on an end-of-year well awarded vacation. But coming back, I have noticed that now I have a lot of automatically generated reports failing.  Common point was they were scheduled to be ran at the same time. So I reviewed them, spanned other the time to avoid the Commserve’s CPU and DB crush. It went a bit better, but still failed.

So I dug deeper, and noticed some reports got stuck in the job queue. Looking at the webconsole.log file, here’s what I could find : 

1014  ERROR ExportHandler:runAndGetPhantomJSStatus:1004 - Failed to run PhantomJS: 

 java.lang.RuntimeException: phantom: Process Timed out after 45 minutes

    at commvault.web.reports.ReportsUtil.runProcess(ReportsUtil.java:1185)

    at commvault.web.reports.ExportHandler.runAndGetPhantomJSStatus(ExportHandler.java:1001)

    at commvault.web.reports.ExportHandler.export(ExportHandler.java:741)

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:278)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

3920 7 01/04 15:17:53 CommserveIP 1014  ERROR ExportHandler:export:838 - Export failed. 

 java.lang.RuntimeException: phantom: Process Timed out after 45 minutes

    at commvault.web.reports.ReportsUtil.runProcess(ReportsUtil.java:1185)

    at commvault.web.reports.ExportHandler.runAndGetPhantomJSStatus(ExportHandler.java:1001)

    at commvault.web.reports.ExportHandler.export(ExportHandler.java:741)

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:278)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

3920 7 01/04 15:17:53 CommserveIP 1014  ERROR ExportHandler:handleRequest:320 - 

 java.lang.RuntimeException: Error while exporting

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:298)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

3920 7 01/04 15:17:53 CommserveIP 1014  ERROR MainController:processRequest:133 - MainController failed to process target runReportSchedule.do. IncidentId is: phtx6jisz3mzmlbyku. Exception is: java.lang.reflect.InvocationTargetException

 java.lang.RuntimeException: Error while exporting

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:298)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

3920 7 01/04 15:17:53 CommserveIP 1014  ERROR ScheduleFilter:doFilter:219 - JOBID: 4245158. 

 java.lang.RuntimeException: Error while exporting

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:298)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

Wrapped by: java.lang.RuntimeException: java.lang.RuntimeException: Error while exporting

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:323)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

Wrapped by: java.lang.reflect.InvocationTargetException: null

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.core.sso.NegotiateSecurityFilter.doFilter(NegotiateSecurityFilter.java:164)

Wrapped by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

    at commvault.web.core.MainController.processRequest(MainController.java:208)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.core.sso.NegotiateSecurityFilter.doFilter(NegotiateSecurityFilter.java:164)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.core.blockapplicationservice.BlockApplicationsFilter.doFilter(BlockApplicationsFilter.java:127)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

3920 7 01/04 15:17:54 CommserveIP 1014  WARN  ServerBroker:getResponseAndHandleExceptions:464 - Invalid response returned: 401

 3920 7 01/04 15:17:54 CommserveIP 1014  ERROR ScheduleFilter:doFilter:244 - JOBID: 4245158. Error while invalidating the token and session after running schedule UNAUTHORIZED_REQUEST

 

CommserveIP replaces my Commserve’s IP, but the rest is plain.

 

This error example is taken from a generation of a filtered AuxCopyJob report I tried to Save as PDF, but to HTML or else is the same, even to send through mail to any format.

The report is viewable on the screen, from the webconsole. But each time I try to save/download/send the report in any form, a PhantomJS process is initiated on the Commserve, grabbing CPU until I kill it : 

And the more reports I request, the more PhantomJS processes I get (each grabbing their CPU share until my CS is almost dying and we have to restart services).

 

I found no hotfixes since fr24 HP29, so before raising a case to support, I wanted to know if some other people is experiencing such, more or less.

Thanks for your feedback, even if you’re in that FR/HP version and your reports are working fine. :wink:

icon

Best answer by Shane 26 January 2022, 05:11

View original

17 replies

S
Rank
Userlevel 7
Badge +15

Hi @Laurent 

When I check case history for this error:

ERROR ExportHandler:runAndGetPhantomJSStatus:1004 - Failed to run PhantomJS: 
 java.lang.RuntimeException: phantom: Process Timed out after 45 minutes

We do have a few examples with various resolutions, so I’m separating this from the Log4J updates at this point.

In some cases the errors are due to a custom report where there are some custom column widths, which result in total column widths larger than the maximum allowed.

Aux Copy Report does feature in a few examples and scheduled reports have been resolved by removing the original schedule and rescheduling - I doubt this will help with your example as manual report export is also failing.

There are a few older (more than 1 year) examples where Development have provided updated report templates, but since you’re on a much more recent version than the affected historic versions, I’m going to rule these out for now.

One symptom that is often seen is some report formats are failing, but csv is successful - would you be able to test a CSV formatted report please?

Thanks,

Stuart

 

Customer Support
L
Rank
Userlevel 6
Badge +15

Hi @Stuart Painter and thanks for this very interesting feedback !

Some of the reports had custom column width (because information is often unreadable in a 10chars column :smile: ), but some other were standard.

I am wondering if, one or two report job(s) (with custom column widths) in the job queue would cause the next scheduled reports stucked then fail, until concerned services are cycled (or phantomJS processes are killed) ? 

Yesterday my scheduled reports failed, while the same reports today have been successfully generated (and sent through mail).

Another intersting point you raised is report versions. For the ‘out of the box’ reports, they’re up to date. But I have downloaded through Store some custom reports, and maybe some of them are outdated, I mean, use ‘deprecated features’ and would require to be rearranged/updated to not cause potential issues.

I will retry my custom report generation and check if CSV export works when PDF or HTML fails, and will let you know later. 

L
Rank
Userlevel 6
Badge +15

Hi ! Gentle followup on this topic. Today again, all the reports were properly generated. So far, so good.

I just don’t know where to dig for forensics yet, but I’m trying not to customize the reports.

I’ll wait until next wednesday where I have many big reports in the queue, to check if all is now magically solved, or not :blush:

S
Rank
Userlevel 7
Badge +15

Hi @Laurent 

Thanks for the update, did CSV generated reports work?

If you have a large run of reports, it would be worth increasing file versions for ScheduledReportingService.log and schedulereport.log so that we capture log messages during the timeframe. Let’s try file versions = 50 for both those logs.

Please be prepared (or schedule a job) to collect logs following the report run so these can be reviewed later.

This issue may have been caused by a hung process that has then affected subsequent jobs, but now that process has been released, new jobs are running as expected.

Without logs for the affected timeframe, I’m just guessing.

Let’s see how the large scheduled report run performs and go from there.

Thanks,

Stuart

Customer Support
L
Rank
Userlevel 6
Badge +15

Hi @Stuart Painter 

Yes, CSV reports worked fine, and my other reports still work fine so far. 

Thanks for highlighting the proper logfiles to look at for troubleshooting :thumbsup:

Commvault provide so many logfiles that in the urge, I did not really know where to look to troubleshoot. :smiley:

 

Next time the report fails, I’ll search into those 2. I was thinking that it could be related to tomcat, so webserver logs to look at, also. 

S
Rank
Userlevel 7
Badge +15

Hi @Laurent 

Yes, you’re correct, there’s a lot of connected processes and webconsole.log (Tomcat) and webserver.log (IIS web server process) may show some messages, which is why support usually request a full logset to review for the full picture.

Also a Send Log Files job by default will collect windows application and system logs or for unix /var/log/messages to give us OS level messages.

But the main processes for this particular issue are the schedule report logs.

Keep me posted on the next report run results.

Thanks,

Stuart

Customer Support
L
Rank
Userlevel 6
Badge +15

Hi @Stuart Painter 

So, this morning we have many reports at 8:00AM beeing generated, and some failed.

I took one as a sample and reviewed through the logs what I could fetch for it : 

7992  574c  01/10 08:01:03 4259482 genReportsWebSchedule::downloadReportFile() - Download error message [0xEDD00000:{CvCURL::CvProtocolReq::download(462)} + {CvCURL::CvProtocolReq::SendReqest(613)} + {CvCURL::CvProtocolReq::Execute(689)/CvCURL.0.0x0-Requested Operation Failed. Return value from CURL: [0] Response code from server: [903]}]

7992  574c  01/10 08:01:03 4259482 genReportsWebSchedule::generateReport() - Failed to generate report using any web console.

7992  574c  01/10 08:01:03 4259482 EvReportMgr::GenerateReportNonSecurity() - Failed to generate report for report type=[8012] task ID=[4259482].

7992  574c  01/10 08:01:03 4259482 EvReportMgr::disposeReport() - Backup job summary : Report Failed

7992  574c  01/10 08:01:03 4259482 genReportBase::getReportUrl() - Generating report using web console [http://MY-COMMSERVE:80/webconsole].

7992  574c  01/10 08:01:04 4259482 LIBCURL::CvInternetGateway::isGatewayClientValid() - Internet Gateway Client Id [4602]:

7992  574c  01/10 08:01:04 4259482 CvInternetGateway::initialize() - JobType not DOWNLOAD_UPDATES, Initializing Direct Connection

7992  574c  01/10 08:01:04 4259482 mailRpt::sendMailThroughTemplateFeature() - Calling SendEmailTemplateMail with UserId [1] LocaleId [0] TokenXML:[<?xml version="1.0" encoding="UTF-8" standalone="no" ?><Api_SendEmailTemplateMailRequest customWrapper="&lt;!doctype html>&lt;html>&lt;head>&lt;title>Message&lt;/title>&lt;/head>&lt;body>&lt;div style=&quot;width: auto; padding : 0; margin: 0 auto; border : 0; &quot;>&lt;table border=&quot;0&quot; cellpadding=&quot;0&quot; cellspacing=&quot;0&quot; bgcolor=&quot;#ffffff&quot; style=&quot;color: #666; background: #ffffff; border : 0; &quot;>&lt;tbody>&lt;tr>&lt;td width=&quot;100 % &quot; valign=&quot;middle&quot; bgcolor=&quot;#ffffff&quot; style=&quot;color: #666666; background: #ffffff; vertical - align: middle; font - size: 1em; line - height: 1.2em; text - align: left; font - family: Helvetica, Arial, sans - serif; &quot;>[CONTENT_EMAIL]&lt;/td>&lt;/tr>&lt;/tbody> &lt;/table>&lt;/div>&lt;/body>&lt;/html>" isReport="1" templateTypeId="14"><tokenValues name="WEB_REPORT_URL" value=""/><tokenValues name="WEB_REPORT_FAILURE_STRING" value="&lt;br>&lt;a href = &quot;http://MY-COMMSERVE:80/adminconsole/#/reportDetails?reportId=121&amp;Table1527261447810.hide=RetainDataUntil,Client,Agent,OperationType,VMDatastore,VMOperatingSystem,ThroughputGBHour&amp;Table1527261447810.show=StartTime,EndTime&amp;view=uview-MYCOMPANYNLLast24hSummary&amp; &quot;style = &quot;color:#2f4e66; font-family:Tahoma,verdana,arial,sans-serif; font-size: 24px;&quot;>Backup job summary&lt;/a>&lt;br>&lt;div style = &quot;margin-top:10px; margin-bottom:10px&quot;>&lt;font face=&quot;Tahoma&quot; size= &quot;2&quot; style=&quot; color:#7d7c7c &quot;>Generated by schedule:  &lt;/font>&lt;font face=&quot;Tahoma&quot; size= &quot;2&quot;  >&lt;a href = &quot;http://MY-COMMSERVE:80/adminconsole/#/reportScheduleDetails/870102/870230?nav=ReportSchedules&quot;style = &quot;color:#2f4e66;&quot;>MYCOMPANY.NL-Last24hSummary&lt;/a>&lt;/font>&lt;br>&lt;font face=&quot;Tahoma&quot; size= &quot;2&quot; style=&quot; color:#7d7c7c &quot;>Status: &lt;/font>&lt;font face=&quot;Tahoma&quot; size= &quot;2&quot;  style=&quot;color:#fc0000;&quot; >Report Failed&lt;/font>&lt;br>&lt;font face=&quot;Tahoma&quot; size= &quot;2&quot; style=&quot; color:#7d7c7c &quot;>Job ID: &lt;/font>&lt;font face=&quot;Tahoma&quot; size= &quot;2&quot;  > &lt;a href = &quot;http://MY-COMMSERVE:80/adminconsole/#/jobs/4259482&quot; style=&quot;color:#2f4e66;&quot;>4259482&lt;/a>&lt;/font>&lt;/div>"/><tokenValues name="WEB_REPORT_UNSUBSCRIBE" value=""/><tokenValues name="WEB_REPORT_SCHEDULE_NAME" value=""/><tokenValues name="WEB_REPORT_SUBJECT" value="Backup job summary"/><toEmails val="myemail@myaddress.fr"/><toEmails val="helpdesk@myemail.nl"/><toEmails val="simpana@groups.mycompany.com"/><attachments attachmentName="D:\CV\Reports\MYCOMPANY.NL-Last24hSummary.pdf" attachmentPath="D:\CV\Reports\MYCOMPANY.NL-Last24hSummary.pdf"/></Api_SendEmailTemplateMailRequest>]

7992  574c  01/10 08:01:05 4259482 mailRpt::sendMailThroughTemplateFeature() - Email template not found.

7992  574c  01/10 08:01:05 4259482 REPORT COMPLETED, Job ID = 4259482

 

Well, I think all the ‘Email template not found’ are ‘normal’, as this looks to be logged for each report. But the  ‘903’  response code by server looks to be a good path to follow.

And in the webConsole.log file, I could find this : 

3920 149 01/10 08:01:03 172.16.10.99 1014  ERROR MainController:processRequest:133 - MainController failed to process target runReportSchedule.do. IncidentId is: pi4b670bz20dzfi9oaq. Exception is: java.lang.reflect.InvocationTargetException

 java.lang.RuntimeException: Error while exporting

    at commvault.web.reports.ExportHandler.handleRequest(ExportHandler.java:298)

    at commvault.web.reports.ExportHandler.handleRequestForRunReportSchedule(ExportHandler.java:165)

    at jdk.internal.reflect.GeneratedMethodAccessor394.invoke(Unknown Source)

    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base/java.lang.reflect.Method.invoke(Unknown Source)

    at commvault.web.core.MainController.processRequest(MainController.java:103)

    at commvault.web.core.MainController.doGet(MainController.java:296)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ReportsFilter.doFilter(ReportsFilter.java:298)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

    at commvault.web.reports.ExportFilter.doFilter(ExportFilter.java:76)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

3920 149 01/10 08:01:03 172.16.10.99 1014  ERROR ScheduleFilter:doFilter:219 - JOBID: 4259482. 

--

And there I’m stuck.

So, it looks like I should open a case to support because they probably read those logs in a much more efficient way that I do :smile:  

H
Rank
Badge

Hello @Laurent, 

Please open a ticket so we can understand the issue better and investigate it.

Thanks,

Hari

S
Rank
Userlevel 7
Badge +15

Hi @Laurent

As suggested, please open a support case and PM me the case number so I can track and then add the solution here for the benefit of the Community.

Thanks,

Stuart

Customer Support
S
Rank
Userlevel 3
Badge +8

Hi @Laurent

As suggested, please open a support case and PM me the case number so I can track and then add the solution here for the benefit of the Community.

Thanks,

Stuart

I have a similar issue since 11.26.3 on multiple CommCells.

Scheduled or Manual email Web reports fail, but downloads work.

Getting a bit of a runaround from Support with no clear resolution in sight.

Could this be related to this thread?

S
Rank
Userlevel 7
Badge +15

Hi @Shane 

Thanks for the case PM, I’ll take a look for you internally.

Thanks,

Stuart

Customer Support
L
Rank
Userlevel 6
Badge +15

Hi there, just to update, that since last week I had no report generation failures, but I also had to reboot my Commserve after posting last time, so I don’t feel saved yet.. :no_mouth:  When this occurs again I’ll log a case and reach you Stuart, to maybe help troubleshooting.

S
Rank
Userlevel 3
Badge +8

Hi @Shane 

Thanks for the case PM, I’ll take a look for you internally.

Thanks,

Stuart

Resolved!
The Engineer added SecurityProtocol with a value of ‘0’ to the CS’s Additional Settings, recycled the services and the emailed reports are working!

https://documentation.commvault.com/additionalsetting/details?name=SecurityProtocol

S
Rank
Userlevel 7
Badge +15

Hi @Shane 

I’m pleased to hear you have a resolution!

 

@Laurent 

Finding it difficult to mark a single post as “Best answer” to close this thread off, so selected the solution as provided to Shane. If you do see a recurrence, please reach out so we can take a closer look at the issues you’re seeing.

 

Thanks,

Stuart

Customer Support
L
Rank
Userlevel 6
Badge +15

No problem Stuart. I have looked at the provided link and switching to ‘0’ would disable SSO for webconsole… 

..but without that much explanation or scope details, so I would not implement this..  

As soon as I get another occurence of this issue I’ll raise a case and update here.

Thanks !

M
Rank
Badge +1

Hi,

Same issue.

Several report executions, several phantomjs processes.

So, I fixed property

...and restarted “all services” (process manager)

Same issue :(

Only with PDF and HTML format... CSV is OK.

L
Rank
Userlevel 6
Badge +15

Woops, I forgot to mention that this had been fixed by applying the latest maintenance release (or hotfix pack) on my Commserve, that included the hotfix (4627) for this issue. 

Reply


Terms & ConditionsCookie settings