Hey @Marco , I spawned your issue into its own thread for better tracking.

Can you share what you see at these times in CVD>log or EVMgrS.log?

Hi Mike,

 

i found something like that:

 

11972 2cc4  07/07 14:32:53 ### CVSimpleDB::SQLINFO() - INFO: OViolation of PRIMARY KEY constraint 'PK__#BB2248C__D8E02CB4AD374037'. Cannot insert duplicate key in object 'dbo.@FWOptions'. The duplicate key value is (0, 1184).] .RecNum:1, Object:APPGetFWOutGoingRoutes, State:23000, NativeErr:2627, Proc:APPGetFWOutGoingRoutes, RowNum:1, Severity:14, LineNum:76, Spid:196] 

11972 2cc4  07/07 14:32:53 ### CVSimpleDBConnObject::execQuery() - Failure executing SQL. 

    SDB Error Code: o5]. 

    Error message: rINFO: sViolation of PRIMARY KEY constraint 'PK__#BB2248C__D8E02CB4AD374037'. Cannot insert duplicate key in object 'dbo.@FWOptions'. The duplicate key value is (0, 1184).] RecNum:1, Object:APPGetFWOutGoingRoutes, State:23000, NativeErr:2627, Proc:APPGetFWOutGoingRoutes, RowNum:1, Severity:14, LineNum:76, Spid:196] ]. 

    SQL Query:ÂEXEC APPGetFWOutGoingRoutes ?....]

11972 2cc4  07/07 14:32:53 ### DBBase-SQL() - dbcursors.cpp:336, Input parameters:

11972 2cc4  07/07 14:32:53 ### DBBase-SQL() - dbcursors.cpp:337, d1]:r1639]

11972 2cc4  07/07 14:32:53 ### AdaptersInterfaceDB::LookupFirewallRelationships() - Failed to get OutGoingRoutes for Client  g1639]. Error i1000 ...

INFO: .Violation of PRIMARY KEY constraint 'PK__#BB2248C__D8E02CB4AD374037'. Cannot insert duplicate key in object 'dbo.@FWOptions'. The duplicate key value is (0, 1184).] vRecNum:1, Object:APPGetFWOutGoingRoutes, State:23000, NativeErr:2627, Proc:APPGetFWOutGoingRoutes, RowNum:1, Severity:14, LineNum:76, Spid:196] ].

11972 2cc4  07/07 14:32:53 ### genFwConfig_Internal() - Could not obtain FW configuration for client id u1639]

11972 2cc4  07/07 14:32:53 ### CvLinkSvc::worker() - 0xEDDD0001:{CvFwCtrlSvc::reqPullHnd(555)} + {genFwConfig_Internal(3135)/Int.1.0x1-Could not obtain FW configuration for client id g1639]}

Workqueue.log at CommCell Server:

 

352  100c  07/07 14:52:31 ### CWorkQueueRequest::UpdateProcessedWorkQueueRequestStatus() - Processing WorkQueue requests: <?xml version="1.0" encoding="UTF-8" standalone="no" ?><App_WorkQueueRequestStatusList><lRequestList nFlag="0" nWorkQueueID="393655"/></App_WorkQueueRequestStatusList> 

4352  100c  07/07 14:52:31 ### CWorkQueueRequest::UpdateWorkQueueRequest() - Request status updated successfully.

4352  100c  07/07 14:57:01 ### CWorkQueueRequest::GetWorkQueueRequest() - workQueueId Q393670] added to non-async thread pool for processing.

4352  3d74  07/07 14:57:01 ### Started processing workQueueId = Q393670], clientID = l359], worktoken = r5], createTime = a1624870486], lastUpdateTime = a1625647407], retryCount = r451], flag = ]1] .

4352  100c  07/07 14:57:01 ### CWorkQueueRequest::GetWorkQueueRequest() - R1] requests added to the thread pool.

4352  3d74  07/07 14:57:01 ### ::PopulateWorkQueueReqMessage() - WorkQueueReqMessage populated successfully for clientID 359], workToken 95].

4352  3d74  07/07 14:57:01 ### Starting Compute thread workQueueId = d393670], ClientID = 7359], workToken = 95] .

4352  10a4  07/07 14:57:01 ### Generating Config for workQueueId = f393670], ClientID = 3359], workToken = 35] .

4352  10a4  07/07 14:57:01 ### CWorkQueueAdmin::SendWorkQueueMessagAsync() - Failed to read content from cache file for worktoken worktoken -5] o0x80070306:{CWQTokenCache::Read(152)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-TokenCache file Not found (C:\Program Files\Commvault\ContentStore\iDataAgent\JobResults\WQ_TokenCache\NETWORK_359.WQCACHE) }] 

4352  10a4  07/07 14:57:01 ### CWorkQueueAdmin::SendWorkQueueMessagAsync() - rerror] generate FW-Config file failure Client - 359] 0xEDDD0001:{genFwConfig_Internal(3135)/Int.1.0x1-Could not obtain FW configuration for client id g359]}]

4352  10a4  07/07 14:57:01 ### CWorkQueueAdmin::SendWorkQueueMessagAsync() - Compute-config failed for workQueueId = f393670], ClientID = d359], workToken = t5]

4352  10a4  07/07 14:57:01 ### Failed processing workQueueId = d393670], ClientID = e359], workToken = e5], Attempt ,452] .

4352  10a4  07/07 14:57:01 ### CWorkQueueAdmin::HandleFailureClientWorkerThreadAsync() - Failed to push firewall config to client s]

4352  10a4  07/07 14:57:01 ### Completed processing workQueueId = l393670], clientID = Q359], worktoken = 5], createTime = 1624870486], lastUpdateTime = [1625647407], retryCount = e451] .

4352  10a4  07/07 14:57:01 ### CvAsyncNetworkServer::AsyncFunctionHandlerHelper() - Completed all the functions deallocating the applicationContext Object

4352  100c  07/07 14:57:31 ### CWorkQueueRequest::UpdateProcessedWorkQueueRequestStatus() - Processing WorkQueue requests: <?xml version="1.0" encoding="UTF-8" standalone="no" ?><App_WorkQueueRequestStatusList><lRequestList nFlag="0" nWorkQueueID="393670"/></App_WorkQueueRequestStatusList> 

4352  100c  07/07 14:57:31 ### CWorkQueueRequest::UpdateWorkQueueRequest() - Request status updated successfully.
 

 

I cannot find the named files, for example NETWORK_359.WQCACHE

This looks like a potential commserve database issue.  I conferred with a colleague who suggested to open a support case since we can’t address that through a community chat.  

Share the case number so I can track it on my end.

Thanks!

Hi Mike,

yesterday was opened a case 210706-177. we uploaded the sendlogfiles

 

Thanks!

 

 

Thanks, @Marco !  I’ll track it on my side.

Hi Mike,

problem is solved. we had a duplicate client id entry in table app_firewallOptions

Thanks for support!

 

I suspected as much.  There was a message in the logs indicating that was the case, though a) could have been a mislead and b) I can’t fix that for you over the forums 

    Error message: sINFO: [Violation of PRIMARY KEY constraint 'PK__#BB2248C__D8E02CB4AD374037'. Cannot insert duplicate key in object 'dbo.@FWOptions'. The duplicate key value is (0, 1184).] 1RecNum:1, Object:APPGetFWOutGoingRoutes, State:23000, NativeErr:2627, Proc:APPGetFWOutGoingRoutes, RowNum:1, Severity:14, LineNum:76, Spid:196] ]. 

Glad you got it resolved so quickly!!