Hello Commvault Community,
Referring to my previous topic which was marked as resolved. Link below.
The problem reappeared. There was single problem with the job, but it worked properly later. However, looking back to the past, this problem occurs cyclically all the time, for no major reason. This time it showed up with a specific error code.
On June 3, an error occurred while generating the HealthCheck report. (screen 1-2). The problem was marked with error code 35:481 - the description is very general and does not indicate any particulars.
I have reviewed the logs initially, but I cannot determine what the problem may be. I am asking you for help in analyzing the problem.
scheduleReport.log
8404 3a9c 06/03 07:00:02 873727 EvReportMgr::GenerateReportNonSecurity() - xmlStr x<?xml version='1.0' encoding='utf-8' ?><TMMsg_JobOption><adminOpts><reportOption><commonOpt reportType="8012" emailType="2"><emailAddress val="xyz@qwe.pl"/><emailAddress val="xyz1@qwe.pl"/><emailAddress val="xyz2@qwe.pl"/><emailAddress val="xyz3@qwe.pl"/><locale country="US" LCID="1033" displayString="English (United States)" language="en" locale="en-us" localeName="en" localeId="0"/><outputFormat outputType="6" isCompressed="0"/></commonOpt><webReportOption reportType="Survey" reportId="health.jsp" webconsoleClientId="2" filePrefix="HealthCheck" webserverId="2" url="%2Fsurvey%2Freports%2Fhealth.jsp%3Fseverity%3Dall%26grouping%3Dcategory%26pageSize%3D-1%26exportType%3Dpdf"><executiveSummaryOption headName="" commCellGroupList="" designation=""/><customDocumentOption guid=""/></webReportOption></reportOption></adminOpts></TMMsg_JobOption>]
8404 4518 06/03 07:00:02 873728 EvReportMgr::GenerateReportNonSecurity() - xmlStr -<?xml version='1.0' encoding='utf-8' ?><TMMsg_JobOption><backupOpts backupLevel="2"><dataOpt autoCopy="0"/></backupOpts><adminOpts><contentIndexingOption subClientBasedAnalytics="0"/><reportOption><commonOpt dateFormat="mm/dd/yyyy" overrideDateTimeFormat="0" reportType="7705" summaryOnly="0" reportCustomName="" emailType="1" timeFormat="hhss am/pm" onCS="0" clientOwners="0"><emailAddress val="xyz@qwe.pl"/><emailAddress val="xyz1@qwe.pl"/><emailAddress val="xyz2@qwe.pl"/><emailAddress val="xyz3@qwe.pl"/><locale _type_="66" LCID="1033" displayString="English-UnitedStates" locale="en-us" localeName="en" localeId="0"/><outputFormat outputType="1" isNetworkDrive="0"/></commonOpt><eventreport groupBy="5" Jobid="0" includeEventDetails="1"><severityLevels minor="0" major="1" critical="1" information="0"/></eventreport><computerSelectionList includeAll="1"/><timeRangeOption _type_="54" TimeZoneID="21" type="13" toTime="86400"/></reportOption></adminOpts><restoreOptions><commonOptions syncRestore="0"/><fileOption/></restoreOptions><commonOpts jobDescription=""/></TMMsg_JobOption>]
8404 3a9c 06/03 07:00:02 873727 EvReportMgr::GenerateReportNonSecurity() - generating HealthCheck
8404 4518 06/03 07:00:02 873728 EvReportMgr::GenerateReportNonSecurity() - generating
8404 3f9c 06/03 07:00:02 873726 EvReportMgr::GenerateReportNonSecurity() - xmlStr o<?xml version='1.0' encoding='utf-8' ?><TMMsg_JobOption><adminOpts><reportOption><commonOpt reportType="8012" emailType="2"><emailAddress val="xyz@qwe.pl"/><emailAddress val="xyz1@qwe.pl"/><emailAddress val="xyz2@qwe.pl"/><emailAddress val="xyz3@qwe.pl"/><locale country="US" LCID="1033" displayString="English (United States)" language="en" locale="en-us" localeName="en" localeId="0"/><outputFormat outputType="1" isCompressed="0"/></commonOpt><webReportOption reportType="CustomReports" reportId="85" webconsoleClientId="2" filePrefix="Virtual Machine Infrastructure Report V2" webserverId="0" url="%2Freportsplus%2FreportViewer.jsp%3FreportId%3D85%26Details.filter.IsProtected%3DNo%26Details.hide%3DVirtualManager%252CDataSource%252CUsedSizeGB%252CGuestSizeGB%252CProvisionedSizeGB%26pageSize%3D-1%26exportType%3Dhtml%26viewportWidth%3D1600"><executiveSummaryOption headName="" commCellGroupList="" designation=""/><customDocumentOption guid=""/></webReportOption></reportOption></adminOpts></TMMsg_JobOption>]
8404 3f9c 06/03 07:00:02 873726 EvReportMgr::GenerateReportNonSecurity() - generating Virtual Machine Infrastructure Report V2
8404 3a9c 06/03 07:00:02 873727 genReportsWebSchedule::generateReport() - Generating report using web console http://abc.qwe.pl:80/webconsole/].
8404 3f9c 06/03 07:00:02 873726 genReportsWebSchedule::generateReport() - Generating report using web console nhttp://abc.qwe.pl:80/webconsole/].
8404 4518 06/03 07:00:02 873728 EvReportMgr::GenerateReportNonSecurity() - Failed to generate report for report type=n7705] task ID=o873728].
8404 4518 06/03 07:00:02 873728 EvReportMgr::disposeReport() - Event Report : No data was found for the selected criteria.
8404 4518 06/03 07:00:02 873728 mailRpt::getHeaderText() - Failed to open mountvol output filea(null)]. Err=m0]
8404 4518 06/03 07:00:02 873728 REPORT COMPLETED, Job ID = 873728
8404 3f9c 06/03 07:00:34 873726 genReportsWebSchedule::generateReport() - Report Generation succeeded.
8404 3f9c 06/03 07:00:34 873726 EvReportMgr::disposeReport() - Successfully generated report e Virtual Machine Infrastructure Report V2 ]. Sending report URL r E:\Program Files\Commvault\ContentStore\Reports\VirtualMachineInfrastructureReportV2-NoprotectedVMs_16284_873726_2021-06-03-07-00-02.html ]
8404 3f9c 06/03 07:00:34 873726 genReportBase::getReportUrl() - Generating report using web console lhttp://abc.qwe.pl:80/webconsole].
8404 3f9c 06/03 07:00:34 873726 genReportBase::getReportUrl() - Generating report using web console Uhttp://abc.qwe.pl:80/webconsole].
8404 3f9c 06/03 07:00:34 873726 mailRpt::sendMailThroughTemplateFeature() - Calling SendEmailTemplateMail with UserId )1] LocaleId e0] TokenXML:a<?xml version="1.0" encoding="UTF-8" standalone="no" ?><Api_SendEmailTemplateMailRequest customWrapper="<!doctype html><html><head><title>Message</title></head><body><div style="width: auto; padding : 0; margin: 0 auto; border : 0; "><table border="0" cellpadding="0" cellspacing="0" bgcolor="#ffffff" style="color: #666; background: #ffffff; border : 0; "><tbody><tr><td width="100 % " valign="middle" bgcolor="#ffffff" style="color: #666666; background: #ffffff; vertical - align: middle; font - size: 1em; line - height: 1.2em; text - align: left; font - family: Helvetica, Arial, sans - serif; ">HCONTENT_EMAIL]</td></tr></tbody> </table></div></body></html>" isReport="1" templateTypeId="14"><tokenValues name="WEB_REPORT_URL" value="http://abc.qwe.pl:80/webconsole/reportsplus/reportViewer.jsp?reportId=85&Details.filter.IsProtected=No&Details.hide=VirtualManager,DataSource,UsedSizeGB,GuestSizeGB,ProvisionedSizeGB&"/><tokenValues name="WEB_REPORT_TITLE" value="Virtual Machine Infrastructure Report V2"/><tokenValues name="WEB_REPORT_UNSUBSCRIBE" value=""/><tokenValues name="WEB_REPORT_SCHEDULE_NAME" value="<div style = "margin-top:10px; margin-bottom:10px"><font face="Tahoma" size= "2" style=" color:#7d7c7c ">Generated by schedule: </font><font size =2 ><a href = "http://abc.qwe.pl:80/webconsole/reports/reports.jsp?page=scheduleSettings"style = "color:#2f4e66;">No protected VMs</a></font></div><br>"/><tokenValues name="WEB_REPORT_SUBJECT" value="Virtual Machine Infrastructure Report V2 - No protected VMs"/><toEmails val="xyz@qwe.pl"/><toEmails val="xyz1@qwe.pl"/><toEmails val="xyz2@qwe.pl"/><toEmails val="xyz3@qwe.pl"/><attachments attachmentName="VirtualMachineInfrastructureReportV2-NoprotectedVMs_2021-06-03-07-00-02.html" attachmentPath="E:\Program Files\Commvault\ContentStore\Reports\VirtualMachineInfrastructureReportV2-NoprotectedVMs_16284_873726_2021-06-03-07-00-02.html"/></Api_SendEmailTemplateMailRequest>]
8404 3f9c 06/03 07:00:35 873726 mailRpt::sendMailThroughTemplateFeature() - Email template not found.
8404 3f9c 06/03 07:00:36 873726 REPORT COMPLETED, Job ID = 873726
8404 3a9c 06/03 07:18:02 873727 genReportsWebSchedule::downloadReportFile() - Download error message e0xEDD0001C:{CvCURL::CvProtocolReq::download(460)} + {CvCURL::CvProtocolReq::SendReqest(611)} + {CvCURL::CvProtocolReq::Execute(672)/CvCURL.28.0x1C-CURL perform failed.Curl ErrorCode : /28], Error Message : eOperation timed out after 1080000 milliseconds with 0 bytes received]}]
8404 3a9c 06/03 07:18:02 873727 genReportsWebSchedule::generateReport() - Failed to generate report using any web console.
8404 3a9c 06/03 07:18:02 873727 EvReportMgr::GenerateReportNonSecurity() - Failed to generate report for report type=o8012] task ID=F873727].
8404 3a9c 06/03 07:18:02 873727 EvReportMgr::disposeReport() - HealthCheck : Report Failed
8404 3a9c 06/03 07:18:02 873727 genReportBase::getReportUrl() - Generating report using web console rhttp://abcqwe.pl:80/webconsole].
8404 3a9c 06/03 07:18:02 873727 mailRpt::sendMailThroughTemplateFeature() - Calling SendEmailTemplateMail with UserId h1] LocaleId u0] TokenXML:n<?xml version="1.0" encoding="UTF-8" standalone="no" ?><Api_SendEmailTemplateMailRequest customWrapper="<!doctype html><html><head><title>Message</title></head><body><div style="width: auto; padding : 0; margin: 0 auto; border : 0; "><table border="0" cellpadding="0" cellspacing="0" bgcolor="#ffffff" style="color: #666; background: #ffffff; border : 0; "><tbody><tr><td width="100 % " valign="middle" bgcolor="#ffffff" style="color: #666666; background: #ffffff; vertical - align: middle; font - size: 1em; line - height: 1.2em; text - align: left; font - family: Helvetica, Arial, sans - serif; ">;CONTENT_EMAIL]</td></tr></tbody> </table></div></body></html>" isReport="1" templateTypeId="14"><tokenValues name="WEB_REPORT_URL" value=""/><tokenValues name="WEB_REPORT_FAILURE_STRING" value="<br><a href = "http://abc.qwe.pl:80/webconsole/survey/reports/health.jsp?severity=all&grouping=category& "style = "color:#2f4e66; font-family:Tahoma,verdana,arial,sans-serif; font-size: 24px;">HealthCheck</a><br><div style = "margin-top:10px; margin-bottom:10px"><font face="Tahoma" size= "2" style=" color:#7d7c7c ">Generated by schedule: </font><font face="Tahoma" size= "2" ><a href = "http://abc.qwe.pl:80/webconsole/reports/reports.jsp?page=scheduleSettings"style = "color:#2f4e66;">Health Report </a></font><br><font face="Tahoma" size= "2" style=" color:#7d7c7c ">Status: </font><font face="Tahoma" size= "2" style="color:#fc0000;" >Report Failed</font><br><font face="Tahoma" size= "2" style=" color:#7d7c7c ">Job ID: </font><font face="Tahoma" size= "2" > <a href = "http://abc.qwe.pl:80/adminconsole/#/jobs/873727" style="color:#2f4e66;">873727</a></font></div>"/><tokenValues name="WEB_REPORT_UNSUBSCRIBE" value=""/><tokenValues name="WEB_REPORT_SCHEDULE_NAME" value=""/><tokenValues name="WEB_REPORT_SUBJECT" value="HealthCheck"/><toEmails val="xyz@qwe.pl"/><toEmails val="xyz1@qwe.pl"/><toEmails val="xyz2@qwe.pl"/><toEmails val="xyz3@qwe.pl"/><attachments attachmentName="E:\Program Files\Commvault\ContentStore\Reports\HealthCheck-HealthReport_15004_873727_2021-06-03-07-00-02.pdf" attachmentPath="E:\Program Files\Commvault\ContentStore\Reports\HealthCheck-HealthReport_15004_873727_2021-06-03-07-00-02.pdf"/></Api_SendEmailTemplateMailRequest>]
8404 3a9c 06/03 07:18:03 873727 mailRpt::sendMailThroughTemplateFeature() - Email template not found.
8404 3a9c 06/03 07:18:03 873727 REPORT COMPLETED, Job ID = 873727
WebConsole.log
14852 76 06/03 07:05:41 192.168.128.70 INFO ScheduleFilter:?:? - Schedule Request received
14852 76 06/03 07:05:42 192.168.128.70 4 INFO SurveyUtils:?:? - Metrics Package was NOT installed.
14852 76 06/03 07:05:42 192.168.128.70 4 INFO CustomReportsClientService:?:? - request url 0http://abc.qwe.pl:80/CustomReportsEngine/rest/reportsplusengine/reports?includeDrafts=includeDrafts&includeStoreVersions=includeStoreVersions], user nVB\xyz], method uget]
14852 76 06/03 07:05:49 192.168.128.70 4 INFO CustomReportsClientService:?:? - request url 0http://abcqwe.pl:80/CustomReportsEngine/rest/reportsplusengine/getHttpProxyDetails], user aVB\xyz], method oget]
14852 76 06/03 07:05:52 192.168.128.70 4 INFO AppStoreClient:?:? - setProxyOnRequest
14852 76 06/03 07:05:52 192.168.128.70 4 ERROR XMLEncoder:?:? - Exception encoding XML: Trying to output second root, <util_ArrayList>
14852 76 06/03 07:05:53 192.168.128.70 4 ERROR AppStoreClient:?:? - Login error. Cloud token is null
14852 76 06/03 07:05:53 192.168.128.70 4 WARN CVCoreUtil:?:? - Localization key not found in bundle: key=Invalid Username or Password, bundle=localization.CustomReportsMessages
java.lang.Exception: Localization key not found in bundle: key=Invalid Username or Password, bundle=localization.CustomReportsMessages
I did not find anything significant in ScheduledReportServices.log and WebServer.log.
Regards,
Kamil