Solved

Replacing old CommServe VM (WindowsServer 2012R2 with a new VM (Windows 2022)

  • 30 September 2022
  • 18 replies
  • 773 views

Userlevel 4
Badge +15

Hello, 

we would like to reconstruct our environment. In addition to the current environment, we plan to build it with new VM's (CommServe, WebServer (Commandcenter) in parallel. Since we currently have only one license in the current environment, the question would be whether we can simply transfer this to the new VM and with the help of a DR backup also all previously made settings ?
We need to replace the VM because the operating system is too old.

Kind Regards

Thomas

icon

Best answer by thomas.S 20 October 2022, 06:57

View original

18 replies

Userlevel 7
Badge +19

Hi @thomas.S,

Yes, this is perfectly possible and it is also documented → https://documentation.commvault.com/2022e/expert/4739_commserve_hardware_refresh_overview.html

The license will travel all along with you with the DR recovery as long as the IP config is dynamic see → 

 

Onno

Please make sure your license IP Address is 255.255.255.255.  If it is, you are good.  If not, please contact your account team to get updated license key.

Userlevel 4
Badge +15

Hello @Onno van den Berg , 

ok, I installed the CommServe on a new VM and deployed a DR backup of the old CommServe. The services on the old system have been shut down. 
When trying to import the DB again during the installation process, it aborts. It seems to have a problem with the HistoryDB.
Since the new CommServe has a new name, I selected Stage. 

@MNRunner , yes, the IP is 255.255.255.255.

Kind Regards

Thomas

Userlevel 7
Badge +16

You need the production recovery, it will adjust the commserve name as well with this option.

Userlevel 4
Badge +15

Hello, 

we have already tried that. I have added the log file and screenshots here.
Maybe I am missing something here. 

Userlevel 7
Badge +16

Hi @thomas.S 

Can you please provide this log file?

C:\Program Files\Commvault\ContentStore\Log Files\DatabaseUpgrade.log

Userlevel 4
Badge +15

Hi @Jos Meijer 
Ok, I have looked into the log in advance. The problem is "DBUpgrade::runUpgrade() - Current SP [28.0.4489590] is higher than SP to be installed[24.0.3589820]. It is not supported for upgrades." But we have version 11.28.8 on the old CommServe. It used to have SP24 but was then updated to SP28. 

Userlevel 7
Badge +16

The interpretation is the other way around.
The log states that you have a 11.28 DB (as you also confirm regarding the upgrade performed) and the installation used is based on 11.24:

10/04 11:50:42 Media Version:[11.80.240.0], Feature Release:[24]

 

Can you use a 2022E (11.28) installer and try again?
Should work when using this version.

Userlevel 4
Badge +15

Everything has worked so far. Also the new name could be made known to the rest of the clients. Only the data is no longer delivered to the web server, which runs on its own VM.

Userlevel 7
Badge +16

Are you operating on the 11.24 install or the 11.28 install?

If still on the same version I am wondering if your installation has been fully completed as the log states the install cannot be completed due to failing steps:

24456 11    10/04 16:17:10 ### ### ### - Error: Failed to complete Main upgrade for database HistoryDB.24456 11    10/04 16:17:10 ### ### ### - Error: Failed to perform main upgrade24456 11    10/04 16:17:10 ### ### ### - Error: Failed to do local configuration for CommServe

 

If you are on 11.28 and the install has completed succesfully, then we need to focus on the logs for communication with the webserver.

Please let us know which scenario applies to your current situation.

Userlevel 4
Badge +15

 

We had already updated the old CommServe to 11.28 a few months ago. The only problem was that my colleague had tried to install on the new VM using the 11.24 installer. After the hint here we had performed the installation again and that also successfully. 
Now we have only one problem with the WebServer. This shows during the readiness check and and when I try to access CommandCenter from the separate VM, the message "Please wait while our services are started. This could take a few minutes."

MongoDB Not Ready.
  • ErrorCode: [587207206]; ErrorMessage: [Failed to execute mongoDB readiness API with response code from server: [500]]. For more information, see KB article.

I suspect it is a firewall problem, since it cannot communicate over port 67688, 1434. 
Since the command center is also installed on the CommServe, I assume a communication problem. When we have solved the problem, Tomcat will be disabled on CommServe. 

 

Userlevel 7
Badge +16

Good to hear the installation worked, regarding the webserver you might want to take a look at this instruction. Originally it is for Commserve LiveSync, but it can also be used if there is a network topology configured between the commserve and a webserver. This re-routes the SQL communication over the Commvault tunnel so you don’t have to open additional ports for SQL in the firewall.

Not necessary, depends on your preferences.

 

Userlevel 4
Badge +15

Hello @Jos Meijer

I believe that it is not a firewall problem. 
My colleague had performed the migration (DR backup) to a new machine. We did not use a local user for the SQL database. I guess that the WebServer now can't communicate with the CommServe because on the WebServer my user is used for the SQL database and on the CommServe probably the account of the person who set it up is used. Am I right with my assumption ?
I can't log in with my user via SQL Studio.

 

Userlevel 7
Badge +16

Hi @thomas.S 

For access to the SQL database the system uses default users created by Commvault as where the passwords are stored in hashed format within the registry.

Your personal account has no influence on the SQL authentication for Commvault functionality.

To find the issue, the answer will most likely be found in the logs of the webserver.
Most likely in cvd.log, cvfwd.log or webserver.log

Userlevel 4
Badge +15

Hello @Jos Meijer 

I found the following in cvfwd.log:
8952 2618 10/12 16:50:21 ######## CN:06266 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:50:48 ######## ######## Starting new (#2/4) SSL worker thread
8952 2618 10/12 16:50:57 VT:00008 CN:06268 [local:cvd<-cvmatapp02] Completed client connection {db2e843a-98b5-8c7c-82f7-6e49433b8599} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:50:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:50:59 ######## CN:06267 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:05 VT:00003 CN:06270 [local:MediaManager<-cvmatapp05] Completed client connection {3c704d56-2d75-4b67-3294-ab83e6879f1e} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:05 VT:00003 CN:06271 [local:MediaManager<-cvmatapp05] Completed client connection {63e14651-5314-d1d7-c539-176375f61dc6} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:23 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:23 ######## CN:06269 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:31 VT:00008 CN:06272 [local:MediaManager<-cvmatapp02] Completed client connection {c0327a14-0f97-778a-ea9d-f7ff060428e2} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:31 VT:00008 CN:06273 [local:MediaManager<-cvmatapp02] Completed client connection {8aa8a1f1-2576-b282-aed8-81ce63ff49cf} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:57 VT:00008 CN:06275 [local:cvd<-cvmatapp02] Completed client connection {3ebc38dc-4df1-5979-8f6e-59b189fa520a} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:51:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:59 ######## CN:06274 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:22 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:22 ######## CN:06276 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:57 VT:00008 CN:06278 [local:cvd<-cvmatapp02] Completed client connection {d81da6a8-c94c-b5e1-9f48-0ababa0dc401} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:52:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:59 ######## CN:06277 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:08 VT:00014 CN:06280 [local:AppManager netw<-cvmatapp01] Completed client connection {37628e9b-fb88-1aa2-e599-79d348c5c1ed} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:10 VT:00014 CN:06281 [local:AppManager netw<-cvmatapp01] Completed client connection {499856b7-0f33-bc6c-ee33-6bf3ca5905a5} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:21 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:21 ######## CN:06279 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:29 VT:00014 CN:06282 [local:AppManager netw<-cvmatapp01] Completed client connection {95cd1eb9-11a3-f3ef-91c0-98ee48b33463} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:50 VT:00014 CN:06284 [local:AppManager netw<-cvmatapp01] Completed client connection {cae72b98-2b5d-79d2-4f27-d4224103241e} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:57 VT:00008 CN:06285 [local:cvd<-cvmatapp02] Completed client connection {ec659a1f-b1be-1500-5b49-be093c903403} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:53:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:59 ######## CN:06283 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:54:02 OT:00194 ######## [cvcstweb02] Reached maximum number of init attempts for ON_DEMAND control tunnel from "cvcstapp01" to "cvcstweb02" via (ANY, 10.114.146.238)
8952 2618 10/12 16:54:11 VT:00014 CN:06287 [local:AppManager netw<-cvmatapp01] Completed client connection {39246005-e7bb-79c6-e109-084a375d62f2} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:54:32 VT:00014 CN:06288 [local:EvMgrS<-cvmatapp01] Completed client connection {a0f71adc-7610-ff92-eae1-d2cfd6a98813} to C:\Program Files\Commvault\ContentStore\Base\Sockets\EvMgrS
8952 2618 10/12 16:54:33 VT:00014 CN:06289 [local:AppManager netw<-cvmatapp01] Completed client connection {2941107a-6e5d-e217-1538-5d223d3e47e2} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access

Userlevel 7
Badge +16

Hi @thomas.S 

This log entry confirms that the tunnel is using the default TCP 8403 and isn’t able to setup the tunnel properly:

8952 2618 10/12 16:50:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.

 

Can you check:

  • If the process manager on the webserver is showing the correct commserve information?
    To validate if the migration to the new commserve has been performed properly for the webserver.
  • If the webserver can resolve the commserve hostname/ip?
  • If the firewall is allowing for TCP 8403 traffic on both the commserve and the webserver?
Userlevel 4
Badge +15

Hi @Jos Meijer

I went through the points and everything looks okay so far. I wonder what else it could be.

 

 

Userlevel 4
Badge +15

Just for Information, this problem was solved yesterday via remote-session.
https://ma.commvault.com/Case/Details/221019-182

Checked in the logs and the webserver cannot connect to commserve on port 1433
Telnet to CS with port 1433 not working
Opened connection on commserve 1433 Telnet ok
In the logs we can find password encryprion errors
Reconfigured ODBC connections
Webserver/webconsole are working with localhost but not with hostname:
Disabled proxy for local addresses on Edge browser
Connection work also with hostname now.
Thanks for the support. 

Reply