Hello,
Commvault 11.28.115
We have Private Metrics Reporting server installed on separated VM. The VM has Windows Server 2016 and SQL 2016. We need to migrate to newer VMware environment, so I assumed it’s good chance to do software refresh. So I deployed new VM with Windows Server 2022 and my plan was to install Commvault with Metrics Reporting and import DBs from old server and then switch to new server.
First problem was I was unable to import DB because there was some problem with DB upgrade. So I decided to create new DB. But the problem is the webconsole isn’t working. Site is loading and tomcat service after some time stops, or, after restarting IIS, the “Please wait, server are starting...” site appears and that’s it.
I looked into webserver logs and there is 26. error locating server/instance like this:
WebServer:
3128 3 10/30 20:59:47 ### ### dmConf::GetMongoConnectionString - Exception Message oA network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)]
3128 3 10/30 20:59:47 ### ### dmConf::GetMongoConnectionString - Exception Source i.Net SqlClient Data Provider]
3128 3 10/30 20:59:47 ### ### dmConf::GetMongoConnectionString - Exception StackTrace S at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)
at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at DM2WebLib.dmConf.GetMongoConnectionStringFromCS(UInt32 clientId, String instanceName)
at DM2WebLib.dmConf.GetMongoConnectionString(UInt32 clientId, String instanceName)]
So it’s looks like it cannot connect to DB. I checked I believe everything connected to this: firewall (but it’s local DB so..), TCP/IP protocol enabled, IP Addresses and ports in SQL configuration manager, ability to login to SQL managament studio etc. Nothing helped, so I believe it’s connected to errors I see in webconsole log.
There are internal server error 500:
1 10/30 23:07:27 WARN ServerBroker:getResponseAndHandleExceptions:500 - Invalid response returned: 500
1 10/30 23:07:27 ERROR DefaultHandler:sendPollRequest:249 - Web Service error: web service returned an unexpected response
commvault.web.core.net.ServerException: INTERNAL_SERVER_ERROR
at commvault.web.core.net.ServerBroker.getResponseAndHandleExceptions(ServerBroker.java:517)
at commvault.web.core.net.ServerBroker.getServerResponse(ServerBroker.java:179)
at commvault.web.core.net.ServerBroker.getServerResponse(ServerBroker.java:145)
at commvault.web.core.net.ServerBroker.getResponse(ServerBroker.java:247)
at commvault.web.core.net.ServerBroker$Proxy$_$$_WeldClientProxy.getResponse(Unknown Source)
at commvault.web.core.WebServiceMonitor.sendPollRequest(WebServiceMonitor.java:219)
at commvault.web.core.WebServiceMonitor.lambda$pollWebService$0(WebServiceMonitor.java:171)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
1 10/30 23:07:27 ERROR DefaultHandler:sendPollRequest:263 - Web service status: RESPONSE_ERROR
8124 10 10/30 23:07:27 10.105.15.101 ERROR ServerUtil:initializeServerProperties:301 - Failed to load all server properties as an exception was thrown. Web service is not available: state=RESPONSE_ERROR
8 10/30 23:07:29 WARN ServerBroker:getResponseAndHandleExceptions:500 - Invalid response returned: 500
8 10/30 23:07:29 ERROR ServerUtil:getSSOEnabledDomains:448 - Unable to load domain information. INTERNAL_SERVER_ERROR
8 10/30 23:07:29 INFO BlockableApplicationsService:updateSettings:99 - Configuring BlockableApplicationsService
8 10/30 23:07:29 INFO BlockableApplicationsService:loadBlockableApplicationList:175 - Loading application rules file: applications.json
8 10/30 23:07:29 ERROR BlockableApplicationsService:determineAndSetOldInstallation:118 - Failed to determine installed SP due to exception. Assuming new installation.
java.lang.NullPointerException: Cannot invoke "javax.servlet.ServletContext.getAttribute(String)" because "servletContext" is null
at commvault.web.core.ServerUtil.getCsGuid(ServerUtil.java:1666)
at commvault.web.core.ServerUtil.getCsGuid(ServerUtil.java:1662)
at commvault.web.core.ServerUtil.doDummyLogin(ServerUtil.java:1701)
at commvault.web.core.ServerUtil.doDummyLogin(ServerUtil.java:1694)
at commvault.web.core.ServerUtil.prepareRequestWithDummyLogin(ServerUtil.java:796)
at commvault.web.core.util.AppSpecificUtilImpl.getCsInstalledSp(AppSpecificUtilImpl.java:54)
at commvault.web.core.util.AppSpecificUtilImpl$Proxy$_$$_WeldClientProxy.getCsInstalledSp(Unknown Source)
at commvault.web.core.blockapplicationservice.BlockableApplicationsService.getCsInstalledSp(BlockableApplicationsService.java:131)
at commvault.web.core.blockapplicationservice.BlockableApplicationsService.determineAndSetOldInstallation(BlockableApplicationsService.java:116)
at commvault.web.core.blockapplicationservice.BlockableApplicationsService.updateSettings(BlockableApplicationsService.java:104)
at commvault.web.core.blockapplicationservice.BlockableApplicationsService.init(BlockableApplicationsService.java:90)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:83)
at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:66)
at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122)
at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:164)
at org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96)
8124 10 10/30 23:07:29 10.105.15.101 WARN ServerBroker:getResponseAndHandleExceptions:500 - Invalid response returned: 500
I saw some similar topics in Community, so what I did:
- reinstall / repair Commvault instance
- reinstall / repair SQL Server
- reinstall / repair .NET 6.16
- install few different versions of .NET
I checked things described in community topics like this and few others but with no luck:
I will open support case but maybe someone has some ideas.