We are trying to set up a new commvault platform version 11.36.6 under Rocky Linux release 9.1 (Blue Onyx). No error reported during the installation.
But when we tried to access the CommandCenter, below the interface that is shown from the WebBrowser :
After reviewing services status, we found that both WebServerCore and MongoDB are down :
From cvlaunchd.log we can see :
19647 4cbf 08/27 12:11:44 ### checkServices() - WebServerCore is not running. Bring it up!
19647 4cbf 08/27 12:11:44 ### checkServices() - MongoDB is not running. Bring it up!
Trying to bring them up did not work as well.
Any idea why services are not working ?
Kind regards.
Best answer by Sys Engineer
Hi all,
Just to give a brief report of what we’ve done to fix the issue.
We’ve performed a fresh new installation, while disabling SELinux and Firewalld. After the installation, MongoDB service was up and running, but the same issue persisted with WebServerCore service.
After taking a deeper look at the WeServerCore.log file, we found that the service was trying to reach the SQL DB and that the connection was failing. From that, we suspected FQDN resolution and added an entry in /etc/hosts file for the CommServe. Restarted all services and everything worked just fine, CommandCenter is accessible.
If the page shows webserver as running, that means webserver is up & running. Please check CommandCenter.log logs & mail us the exception in CommandCenter.log file.
If the page does NOT show webserver as running, that means dotnet service is crashing. In this case, follow the below steps.
Ssh to webserver machine.
Change directory to WebServerCore.
Set registry sDISABLEASYNC to true in Session.
Set webserver.log level to 10.
Create a temp bash script file e.g. startup.sh with below commands.
#!/bin/sh # This is needed so that we could find which VM we run in CVREG="/etc/CommVaultRegistry" #e.g. "/etc/CommVaultRegistry" INSTANCE="Instance001" #e.g. "Instance001" GALDIR=`grep -w dGALAXYHOME $CVREG/Galaxy/$INSTANCE/Base/.properties | awk '{print $2}'`
. $GALDIR/galaxy_vm #Start dotnet WebServerCore, wait for it to daemonize and to provide PID LD_LIBRARY_PATH=$GALDIR/DotNetCore/shared/Microsoft.NETCore.App/6.0.16:$GALDIR/Base:$LD_LIBRARY_PATH export LD_LIBRARY_PATH ODBCSYSINI=/etc export ODBCSYSINI COREENABLED=$(awk '/\<nEnableCoreDump\>/ { print $2 }' /etc/CommVaultRegistry/Galaxy/Instance001/.properties) if [[ $COREENABLED == 1 ]] then export COMPlus_DbgEnableMiniDump=1 fi $GALDIR/DotNetCore/dotnet $GALDIR/WebServerCore/CVWebServer.dll --interactive --instance $INSTANCE --authentication 3
Execute bash script startup.sh
Try accessing the URL http://<webserver_host_ip>:5000/searchsvc/cvwebservice.svc and let us the exception in ssh window and logs from webserver.log file as well.
MongoDB issue can be handled separately after this one is addressed.
If the page shows webserver as running, that means webserver is up & running. Please check CommandCenter.log logs & mail us the exception in CommandCenter.log file.
If the page does NOT show webserver as running, that means dotnet service is crashing. In this case, follow the below steps.
Ssh to webserver machine.
Change directory to WebServerCore.
Set registry sDISABLEASYNC to true in Session.
Set webserver.log level to 10.
Create a temp bash script file e.g. startup.sh with below commands.
#!/bin/sh # This is needed so that we could find which VM we run in CVREG="/etc/CommVaultRegistry" #e.g. "/etc/CommVaultRegistry" INSTANCE="Instance001" #e.g. "Instance001" GALDIR=`grep -w dGALAXYHOME $CVREG/Galaxy/$INSTANCE/Base/.properties | awk '{print $2}'`
. $GALDIR/galaxy_vm #Start dotnet WebServerCore, wait for it to daemonize and to provide PID LD_LIBRARY_PATH=$GALDIR/DotNetCore/shared/Microsoft.NETCore.App/6.0.16:$GALDIR/Base:$LD_LIBRARY_PATH export LD_LIBRARY_PATH ODBCSYSINI=/etc export ODBCSYSINI COREENABLED=$(awk '/\<nEnableCoreDump\>/ { print $2 }' /etc/CommVaultRegistry/Galaxy/Instance001/.properties) if [[ $COREENABLED == 1 ]] then export COMPlus_DbgEnableMiniDump=1 fi $GALDIR/DotNetCore/dotnet $GALDIR/WebServerCore/CVWebServer.dll --interactive --instance $INSTANCE --authentication 3
Execute bash script startup.sh
Try accessing the URL http://<webserver_host_ip>:5000/searchsvc/cvwebservice.svc and let us the exception in ssh window and logs from webserver.log file as well.
MongoDB issue can be handled separately after this one is addressed.
Did manage to Set registry sDISABLEASYNC to true in Session. But, unfortunately, couldn’t find where can I increase the debug level using CLI. Since from SP 34, with new CV Installations, there is no access to CommCell Console (Java Console),
Below the outpu generated by the script : ./startup.sh
Starting the server in [WEBSERVER] mode as part of [Instance001] instance. Use Combined authentication.
Focus set
Configuring Grpc
Setting absolute expiry for key Galaxy\Instance001\EventManager\sDISABLEASYNC
Setting absolute expiry for key Galaxy\Instance001\EventManager\dEVLOGDIR
Setting absolute expiry for key Galaxy\Instance001\EventManager\WebServer_MAXLOGFILESIZE
Setting absolute expiry for key Galaxy\Instance001\EventManager\WebServer_MAXLOGFILEBACKUPS
Setting absolute expiry for key Galaxy\Instance001\EventManager\WebServer_DEBUGLEVEL
Setting absolute expiry for key Galaxy\Instance001\EventManager\logFlushTimeInterval
Setting absolute expiry for key Galaxy\Instance001\EventManager\WEBSERVER_TRACE
Initialization of Grpc client for CppSharpGrpc service is successfully
Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 35 - An internal exception was caught)
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 startingTimestamp)
at System.Net.Dns.GetHostAddresses(String hostNameOrAddress, AddressFamily family)
at Microsoft.Data.SqlClient.SNI.SNITCPHandle.GetHostAddressesSortedByPreference(String serverName, SqlConnectionIPAddressPreference ipPreference)+MoveNext()
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at CV.WebServer.Program.BuildWebHost(String[] args)
If the page shows webserver as running, that means webserver is up & running. Please check CommandCenter.log logs & mail us the exception in CommandCenter.log file.
If the page does NOT show webserver as running, that means dotnet service is crashing. In this case, follow the below steps.
Ssh to webserver machine.
Change directory to WebServerCore.
Set registry sDISABLEASYNC to true in Session.
Set webserver.log level to 10.
Create a temp bash script file e.g. startup.sh with below commands.
#!/bin/sh # This is needed so that we could find which VM we run in CVREG="/etc/CommVaultRegistry" #e.g. "/etc/CommVaultRegistry" INSTANCE="Instance001" #e.g. "Instance001" GALDIR=`grep -w dGALAXYHOME $CVREG/Galaxy/$INSTANCE/Base/.properties | awk '{print $2}'`
. $GALDIR/galaxy_vm #Start dotnet WebServerCore, wait for it to daemonize and to provide PID LD_LIBRARY_PATH=$GALDIR/DotNetCore/shared/Microsoft.NETCore.App/6.0.16:$GALDIR/Base:$LD_LIBRARY_PATH export LD_LIBRARY_PATH ODBCSYSINI=/etc export ODBCSYSINI COREENABLED=$(awk '/\<nEnableCoreDump\>/ { print $2 }' /etc/CommVaultRegistry/Galaxy/Instance001/.properties) if [[ $COREENABLED == 1 ]] then export COMPlus_DbgEnableMiniDump=1 fi $GALDIR/DotNetCore/dotnet $GALDIR/WebServerCore/CVWebServer.dll --interactive --instance $INSTANCE --authentication 3
Execute bash script startup.sh
Try accessing the URL http://<webserver_host_ip>:5000/searchsvc/cvwebservice.svc and let us the exception in ssh window and logs from webserver.log file as well.
MongoDB issue can be handled separately after this one is addressed.
Since the page does not show the web server as running, we want to try the second procedure :
Under /opt/commvault/WebServerCore folder, where can we implement the below changes (which file) ? :
Set registry sDISABLEASYNC to true in Session.
Set webserver.log level to 10.
Kind regards,
Sorry, there is a mistake in the location of the registry. It should be under EventManager and not under Session.The rest is OK - to create and run a batch which should start the webserver or show where it fails.
Just to give a brief report of what we’ve done to fix the issue.
We’ve performed a fresh new installation, while disabling SELinux and Firewalld. After the installation, MongoDB service was up and running, but the same issue persisted with WebServerCore service.
After taking a deeper look at the WeServerCore.log file, we found that the service was trying to reach the SQL DB and that the connection was failing. From that, we suspected FQDN resolution and added an entry in /etc/hosts file for the CommServe. Restarted all services and everything worked just fine, CommandCenter is accessible.
We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.
