[SP44] Commserve ha test failoover and failover failed [Rockey Linux 9]

Could you please to help suggestion,  i try to find what is the root cause but still uncertain.

Hi ​@kitty-yip , there is active CS Replication job running with Job ID “50”, check the Console to see if that job is still running, suspended or has any problem, fix it then start new replication and failover

Thanks,

Abdu

Found this error on standby server. -->  10/09 19:03:24 ### GetFailoverDetailsFromDb() - Failed to get failover settings from DB. Error code [150995030], error [[ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [0x80070057:{CCvNetwork::Attach_RemoteService(1844)} + {CCvNetwork::Attach_RemoteServiceInternal(1791)} + {CCvNetwork::Attach_RemotePort(1482)} + {CCvNetwork::DemangleHostNameToComps(5165)/W32.87.(The parameter is incorrect. (ERROR_INVALID_PARAMETER.87))-Invalid mangled host name:[**8400*8402] passed. Please use mangled host name.}]]

I attach log of Active & Stand by commserve that test ha failed.

on cvfwd.log found error as below

55541 d904 10/22 21:07:05 OT:00073 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.134):
55541 d904 10/22 21:07:05 OT:00073 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=30 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:05 OT:00074 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.136):
55541 d904 10/22 21:07:05 OT:00074 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=31 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:05 ######## CN:00711 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to OATWS-001-CV: Connect failed: Failed to connect socket bsd=30 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:05 ######## CN:00711 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to OATWS-001-CV: Connect failed: Failed to connect socket bsd=31 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:05 ######## CN:00711 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to @@GATEWAY@@: Connect failed: Failed to connect socket bsd=13 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:08 OT:00075 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.134):
55541 d904 10/22 21:07:08 OT:00075 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=31 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:08 OT:00076 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.136):
55541 d904 10/22 21:07:08 OT:00076 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=32 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:08 ######## CN:00712 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to OATWS-001-CV: Connect failed: Failed to connect socket bsd=31 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:08 ######## CN:00712 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to OATWS-001-CV: Connect failed: Failed to connect socket bsd=32 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:08 ######## CN:00712 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to @@GATEWAY@@: Connect failed: Failed to connect socket bsd=13 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:39 OT:00078 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.134):
55541 d904 10/22 21:07:39 OT:00078 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=13 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:39 OT:00079 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.136):
55541 d904 10/22 21:07:39 OT:00079 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=23 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:39 ######## ######## CS is OFFLINE
55541 d904 10/22 21:07:49 OT:00080 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.134):
55541 d904 10/22 21:07:49 OT:00080 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=23 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:49 OT:00081 ######## [OATWS-001-CV] Error encountered for ON_DEMAND automatic tunnel from "OATWS-001-CV_2" to "OATWS-001-CV" via (ANY, 10.11.102.136):
55541 d904 10/22 21:07:49 OT:00081 ######## [OATWS-001-CV] ERROR: process_connect(): Connect failed: Failed to connect socket bsd=24 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:49 ######## CN:00714 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to OATWS-001-CV: Connect failed: Failed to connect socket bsd=23 to 10.11.102.134:8403: Connection refused
55541 d904 10/22 21:07:49 ######## CN:00714 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to OATWS-001-CV: Connect failed: Failed to connect socket bsd=24 to 10.11.102.136:8403: Connection refused
55541 d904 10/22 21:07:49 ######## CN:00714 [local->OATWS-001-CV:AppManager netw] ERROR: Client connection to OATWS-001-CV failed on OATWS-001-CV_2: Error in tunnel to @@GATEWAY@@: Connect failed: Failed to connect socket bsd=13 to 10.11.102.134:8403: Connection refused

Hi ​@kitty-yip , I checked those logs, but they are not enough to define what is the problem/fix -as far as I can tell- 

We would need more logs, like CSRecoveryAssistant.log from both machines

At this point, I would strongly recommend opening a support case to continue this forward.

Also, those log files have sensitive information, please consider removing the log files from the post.

Standby

44733 12382 10/22 20:56:55 ### CVSimpleDB::SQLINFO() - INFO: [Cannot open database "CommServ" requested by the login. The login failed.] [RecNum:1, State:42000, NativeErr:4060, Severity:-1, Spid:-1] 44733 12382 10/22 20:56:55 ### CVSimpleDB::SQLINFO() - INFO: [Login failed for user 'TRUNCATED'.] [RecNum:2, State:28000, NativeErr:18456, Severity:-1, Spid:-1] 44733 12382 10/22 20:56:55 ### CVSimpleDBConnObject::OpenSQLConnectionUsingConnectionString() - Connecting to the database failed - error returned was: INFO: [Cannot open database "CommServ" requested by the login. The login failed.] [RecNum:1, State:42000, NativeErr:4060, Severity:-1, Spid:-1] INFO: [Login failed for user 'TRUNCATED'.] [RecNum:2, State:28000, NativeErr:18456, Severity:-1, Spid:-1] 44733 12382 10/22 20:56:55 ### CVSimpleDBConnObject::OpenSQLConnectionUsingConnectionString() - Connection attempt 2 failed - ConnStr=[Driver={ODBC Driver 18 for SQL Server};Server=OATWS-001-CV2;Database=CommServ;UID=TRUNCATED;PWD=***;MARS_Connection=yes;AnsiNPW=Yes;APP=cvd;TrustServerCertificate=Yes;], RegDSN=[], RegAccess=[pACCESS]

…..
44733 12382 10/22 20:57:01 ### CActiveOperations::ExecuteCSRecoveryAssistant() - Executing command  [sh /opt/commvault/Base/CSRecoveryAssistant.sh] 44733 12382 10/22 20:57:01 ### CvProcess::system() - sh /opt/commvault/Base/CSRecoveryAssistant.sh  -operation Recovery -skipdump  -csfailover -skipupdatercstatus 44733 12382 10/22 20:57:17 ### CvProcess::system() - Command completed with rc=1

44733 12382 10/22 20:57:17 ### CActiveOperations::ExecuteCSRecoveryAssistant() - Error: [Failed to execute command [sh /opt/commvault/Base/CSRecoveryAssistant.sh  -operation Recovery -skipdump  -csfailover -skipupdatercstatus]. ExitCode [1], ErrStr [], ErrOut [Arg[0] = [-operation] Arg[1] = [Recovery] Arg[2] = [-skipdump] Arg[3] = [-csfailover] Arg[4] = [-skipupdatercstatus] CommServe Recovery failed. Check CSRecoveryAssistant.log for details. Press any key to continue ... 1 Failed to run CSRecoveryAssistant. Please check CSRecoveryAssistant.log file for more information. ]]. Please check CSRecoveryAssistant.log from CommServe instance for details.

Thanks,

Abdu