Solved

Data aging Competing with errors.Cross Check oracle clients fails

  • 8 November 2021
  • 10 replies
  • 537 views

Badge +1

Hi Guys,

I have the below error in my environment.

 The below is the results from dataaging logs,

We  are able to cross check backups

Any tips.

icon

Best answer by Chris Hollis 12 November 2021, 01:09

View original

10 replies

Userlevel 3
Badge +8

Hi @Makadi Happines 

 

Not quite sure why this happens, but you should be able to disable the crosscheck during data aging on the Oracle instance..    this way you should be able to release used space if needed. Not really a solution, I know… sorry :-(

 

Kind regards

Rubeck 

 

 

 

Badge +1

Hi @RubeckDK ,

 

Thank you for the response.

Questions,

When disabling crosscheck on server, What is the impact and how  does it affect prunning on the server?

 

Results of disabling crosscheck on the server.

dataaging logs.

371 ::RunClOraAgent() - dataAgentAttach-> HostName=Host..
8728  70c0  10/19 20:19:39 2841371 ::RunClOraAgent() - Attach_RemoteCommand (ClOraAgent) failed:0x80070102:{CCvNetwork::Attach_RemoteCommand(2137)} + {CCvNetwork::Attach_RemoteCommand(2095)} + {CCvNetwork::Accept(1235)} + {CSessionConnectionSocket::Accept(343)} + {CSessionConnectionSocket::CheckDataWaiting(663)/W32.258.(The wait operation timed out. (WAIT_TIMEOUT.258))-Timed out}
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstance() - RunClOraAgent() failed.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstance() - End Oracle CrossCheck for jobId=2841371 CommCellId=2 clientId=2506 clientName=Client instanceId=336, instanceName=Instance jobControl=000000B40BEDBB70. CROSSCHECK_STATUS=FAILURE
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - CrossCheckForOraClientAndInstance() failed for clientId=2506 instanceId=336.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Mark failure for instanceId=336 oraSID=Instance.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Mark failure for client[Client] with clientId=2506.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Exit for jobId=2841371 CommCellId=2 jobControl=000000B40BEDBB70.
8728  70c0  10/19 20:19:39 2841371 APOracleClient::deleteAF() - Some instances of client [2506] will be skipped due to error.
8728  70c0  10/19 20:19:39 2841371 APOracleInst::deleteAF() - Skip instance [336] on client [2506]
8728  70c0  10/19 20:19:39 2841371 APOracleClient::deleteAF() - Client [2506]: Failed for some or all archive files
8728  70c0  10/19 20:19:39 2841371 APOracleCrossCheck::deleteAF() - Failed for some or all archive files
8728  70c0  10/19 20:19:39 2841371 DATA AGING() - JOB [2841371]: Failed to Cross-Check Oracle Clients. 
8728  70c0  10/19 20:19:39 2841371 APSAPCrossCheck::removeDanglingAF() - Archive files from deleted clients and instances are deleted successfully
8728  70c0  10/19 20:19:39 2841371 ArchPruneBase::getSAPCrossCheckInfo() - Total [0] SAP instances to be cross checked

 

 

Userlevel 7
Badge +23

The Oracle Cross Check is there to ensure that what we have in the Oracle job history matches what Oracle is aware of.  This way, if Oracle expects a certain log file to exist, we don’t delete it.

If the cross check fails, that’s almost always on the Oracle side of things and you should check with your DBA to find out why.

You CAN disable the cross check, but you might delete a log file job that Oracle expects to have access to….so it shouldn’t be done without your DBA being consulted.

Badge +1

Hi @Mike Struening ,

I have requested our Oracle DBA  investigate the crosscheck failures on the DB and The archive logs crossscheck had  no errors.

  This is the error i am getting ”Establishing connection to Client services failed during Crosscheck. Make sure this product's Services are running on the client [IP*server*8400*8402] and check network connectivity between client and CommServe.”

Any tips on how i should go about in solving this issue.

Userlevel 4
Badge +8

@Makadi Happines 

Based upon the log cuts that you provided is the specific oracle instance still running on the client. Also you can check the ClOraAgent log on the client. As this is the log that the data aging process will use to process the cross check. 

 

Badge +1

@Mike Struening ,

Checking data aging logs, RunClOraAgent() fails. See logs below

How do I check ClOraAgent logs on a client?

16012 1c54  11/10 18:04:17 2877733 ::RunClOraAgent() - dataAgentAttach-> HostName=1server start...
16012 1c54  11/10 18:06:17 2877733 ::RunClOraAgent() - Attach_RemoteCommand (ClOraAgent) failed:0x80070102:{CCvNetwork::Attach_RemoteCommand(2137)} + {CCvNetwork::Attach_RemoteCommand(2095)} + {CCvNetwork::Accept(1235)} + {CSessionConnectionSocket::Accept(343)} + {CSessionConnectionSocket::CheckDataWaiting(663)/W32.258.(The wait operation timed out. (WAIT_TIMEOUT.258))-Timed out}
16012 1c54  11/10 18:06:17 2877733 OraCrossCheck::CrossCheckForOraClientAndInstance() - RunClOraAgent() failed.
16012 1c54  11/10 18:06:17 2877733 OraCrossCheck::CrossCheckForOraClientAndInstance() - End Oracle CrossCheck for jobId=2877733 CommCellId=2 clientId=2478 clientName=server instanceId=303, instanceName=CC1 jobControl=00000068828EBFE0. CROSSCHECK_STATUS=FAILURE
16012 1c54  11/10 18:06:17 2877733 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - CrossCheckForOraClientAndInstance() failed for clientId=2478 instanceId=303.

Userlevel 4
Badge +8

@Makadi Happines 

You will need to get a remote session to the client and in the commvault logs files directory you will find the file ClOraAgent log. Or you can right click on the client view - log files and select the log file from the list.

Userlevel 5
Badge +10

@Makadi Happines 

Hope you’re well.

Just out of curiosity, did you happen to update to FR23 or FR24 and then notice this issue occurring? 

I only ask as I had a customer report this same issue after update from FR20 to FR24 - I managed to find a fix that was released in the latest maintenance release for it: (11.24.21): https://documentation.commvault.com/11.24/assets/service_pack/updates/11_24_21.htm
 


Reviewing the fix history, issue originated in FR23 (and is fixed in 11.23.35)

https://documentation.commvault.com/11.23/assets/service_pack/updates/11_23_35.htm


If you’ve updated to either of these feature releases, could I ask that you download/install the latest maintenance release that contains this fix and see if this helps?

Otherwise, please carry on with the above suggestions…

 

Regards,
Chris 

Badge

Hi, How i can run crosscheck and delete backup in comvault?

thanks

Userlevel 7
Badge +23

@Debora , this happens when you run Data Aging.

Reply