Replacing old CommServe VM (WindowsServer 2012R2 with a new VM (Windows 2022)

Hi @thomas.S,

Yes, this is perfectly possible and it is also documented → https://documentation.commvault.com/2022e/expert/4739_commserve_hardware_refresh_overview.html

The license will travel all along with you with the DR recovery as long as the IP config is dynamic see → 

 

Onno

Please make sure your license IP Address is 255.255.255.255.  If it is, you are good.  If not, please contact your account team to get updated license key.

Hello @Onno van den Berg , 

ok, I installed the CommServe on a new VM and deployed a DR backup of the old CommServe. The services on the old system have been shut down. 
When trying to import the DB again during the installation process, it aborts. It seems to have a problem with the HistoryDB.
Since the new CommServe has a new name, I selected Stage. 

@MNRunner , yes, the IP is 255.255.255.255.

Kind Regards

Thomas

You need the production recovery, it will adjust the commserve name as well with this option.

Hello, 

we have already tried that. I have added the log file and screenshots here.
Maybe I am missing something here. 

Hi @thomas.S 

Can you please provide this log file?

C:\Program Files\Commvault\ContentStore\Log Files\DatabaseUpgrade.log

Hi @Jos Meijer 
Ok, I have looked into the log in advance. The problem is "DBUpgrade::runUpgrade() - Current SP [28.0.4489590] is higher than SP to be installed[24.0.3589820]. It is not supported for upgrades." But we have version 11.28.8 on the old CommServe. It used to have SP24 but was then updated to SP28. 

The interpretation is the other way around.
The log states that you have a 11.28 DB (as you also confirm regarding the upgrade performed) and the installation used is based on 11.24:

10/04 11:50:42 Media Version:[11.80.240.0], Feature Release:[24]

Can you use a 2022E (11.28) installer and try again?
Should work when using this version.

Everything has worked so far. Also the new name could be made known to the rest of the clients. Only the data is no longer delivered to the web server, which runs on its own VM.

Are you operating on the 11.24 install or the 11.28 install?

If still on the same version I am wondering if your installation has been fully completed as the log states the install cannot be completed due to failing steps:

24456 11    10/04 16:17:10 ### ### ### - Error: Failed to complete Main upgrade for database HistoryDB.24456 11    10/04 16:17:10 ### ### ### - Error: Failed to perform main upgrade24456 11    10/04 16:17:10 ### ### ### - Error: Failed to do local configuration for CommServe

If you are on 11.28 and the install has completed succesfully, then we need to focus on the logs for communication with the webserver.

Please let us know which scenario applies to your current situation.

We had already updated the old CommServe to 11.28 a few months ago. The only problem was that my colleague had tried to install on the new VM using the 11.24 installer. After the hint here we had performed the installation again and that also successfully. 
Now we have only one problem with the WebServer. This shows during the readiness check and and when I try to access CommandCenter from the separate VM, the message "Please wait while our services are started. This could take a few minutes."
. 

I suspect it is a firewall problem, since it cannot communicate over port 67688, 1434. 
Since the command center is also installed on the CommServe, I assume a communication problem. When we have solved the problem, Tomcat will be disabled on CommServe. 

Good to hear the installation worked, regarding the webserver you might want to take a look at this instruction. Originally it is for Commserve LiveSync, but it can also be used if there is a network topology configured between the commserve and a webserver. This re-routes the SQL communication over the Commvault tunnel so you don’t have to open additional ports for SQL in the firewall.

Not necessary, depends on your preferences.

Hello @Jos Meijer, 

I believe that it is not a firewall problem. 
My colleague had performed the migration (DR backup) to a new machine. We did not use a local user for the SQL database. I guess that the WebServer now can't communicate with the CommServe because on the WebServer my user is used for the SQL database and on the CommServe probably the account of the person who set it up is used. Am I right with my assumption ?
I can't log in with my user via SQL Studio.

 

Hi @thomas.S 

For access to the SQL database the system uses default users created by Commvault as where the passwords are stored in hashed format within the registry.

Your personal account has no influence on the SQL authentication for Commvault functionality.

To find the issue, the answer will most likely be found in the logs of the webserver.
Most likely in cvd.log, cvfwd.log or webserver.log

Hello @Jos Meijer 

I found the following in cvfwd.log:
8952 2618 10/12 16:50:21 ######## CN:06266 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:50:48 ######## ######## Starting new (#2/4) SSL worker thread
8952 2618 10/12 16:50:57 VT:00008 CN:06268 [local:cvd<-cvmatapp02] Completed client connection {db2e843a-98b5-8c7c-82f7-6e49433b8599} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:50:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:50:59 ######## CN:06267 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:05 VT:00003 CN:06270 [local:MediaManager<-cvmatapp05] Completed client connection {3c704d56-2d75-4b67-3294-ab83e6879f1e} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:05 VT:00003 CN:06271 [local:MediaManager<-cvmatapp05] Completed client connection {63e14651-5314-d1d7-c539-176375f61dc6} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:23 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:23 ######## CN:06269 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:31 VT:00008 CN:06272 [local:MediaManager<-cvmatapp02] Completed client connection {c0327a14-0f97-778a-ea9d-f7ff060428e2} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:31 VT:00008 CN:06273 [local:MediaManager<-cvmatapp02] Completed client connection {8aa8a1f1-2576-b282-aed8-81ce63ff49cf} to C:\Program Files\Commvault\ContentStore\Base\Sockets\MediaManager
8952 2618 10/12 16:51:57 VT:00008 CN:06275 [local:cvd<-cvmatapp02] Completed client connection {3ebc38dc-4df1-5979-8f6e-59b189fa520a} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:51:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:51:59 ######## CN:06274 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:22 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:22 ######## CN:06276 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:57 VT:00008 CN:06278 [local:cvd<-cvmatapp02] Completed client connection {d81da6a8-c94c-b5e1-9f48-0ababa0dc401} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:52:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:52:59 ######## CN:06277 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:08 VT:00014 CN:06280 [local:AppManager netw<-cvmatapp01] Completed client connection {37628e9b-fb88-1aa2-e599-79d348c5c1ed} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:10 VT:00014 CN:06281 [local:AppManager netw<-cvmatapp01] Completed client connection {499856b7-0f33-bc6c-ee33-6bf3ca5905a5} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:21 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:21 ######## CN:06279 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:29 VT:00014 CN:06282 [local:AppManager netw<-cvmatapp01] Completed client connection {95cd1eb9-11a3-f3ef-91c0-98ee48b33463} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:50 VT:00014 CN:06284 [local:AppManager netw<-cvmatapp01] Completed client connection {cae72b98-2b5d-79d2-4f27-d4224103241e} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:53:57 VT:00008 CN:06285 [local:cvd<-cvmatapp02] Completed client connection {ec659a1f-b1be-1500-5b49-be093c903403} to C:\Program Files\Commvault\ContentStore\Base\Sockets\cvd
8952 2618 10/12 16:53:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:53:59 ######## CN:06283 [local->cvcstweb02:cvd] ERROR: Client connection to cvcstweb02 failed on cvcstapp01: Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.
8952 2618 10/12 16:54:02 OT:00194 ######## [cvcstweb02] Reached maximum number of init attempts for ON_DEMAND control tunnel from "cvcstapp01" to "cvcstweb02" via (ANY, 10.114.146.238)
8952 2618 10/12 16:54:11 VT:00014 CN:06287 [local:AppManager netw<-cvmatapp01] Completed client connection {39246005-e7bb-79c6-e109-084a375d62f2} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access
8952 2618 10/12 16:54:32 VT:00014 CN:06288 [local:EvMgrS<-cvmatapp01] Completed client connection {a0f71adc-7610-ff92-eae1-d2cfd6a98813} to C:\Program Files\Commvault\ContentStore\Base\Sockets\EvMgrS
8952 2618 10/12 16:54:33 VT:00014 CN:06289 [local:AppManager netw<-cvmatapp01] Completed client connection {2941107a-6e5d-e217-1538-5d223d3e47e2} to C:\Program Files\Commvault\ContentStore\Base\Sockets\AppManager network access

Hi @thomas.S 

This log entry confirms that the tunnel is using the default TCP 8403 and isn’t able to setup the tunnel properly:

8952 2618 10/12 16:50:59 OT:00194 ######## [cvcstweb02] ERROR: cvfwd_iot_wait(): Couldn't complete tunnel connection to cvcstweb02 on 10.114.146.238:8403: The semaphore timeout period has expired.

Can you check:

• If the process manager on the webserver is showing the correct commserve information?
  To validate if the migration to the new commserve has been performed properly for the webserver.
• If the webserver can resolve the commserve hostname/ip?
• If the firewall is allowing for TCP 8403 traffic on both the commserve and the webserver?

Hi @Jos Meijer, 

I went through the points and everything looks okay so far. I wonder what else it could be.