Data aging Competing with errors.Cross Check oracle clients fails

Hi @Makadi Happines 

Not quite sure why this happens, but you should be able to disable the crosscheck during data aging on the Oracle instance..    this way you should be able to release used space if needed. Not really a solution, I know… sorry :-(

Kind regards

Rubeck 

Hi @RubeckDK ,

Thank you for the response.

Questions,

When disabling crosscheck on server, What is the impact and how  does it affect prunning on the server?

Results of disabling crosscheck on the server.

dataaging logs.

371 ::RunClOraAgent() - dataAgentAttach-> HostName=Host..
8728  70c0  10/19 20:19:39 2841371 ::RunClOraAgent() - Attach_RemoteCommand (ClOraAgent) failed:0x80070102:{CCvNetwork::Attach_RemoteCommand(2137)} + {CCvNetwork::Attach_RemoteCommand(2095)} + {CCvNetwork::Accept(1235)} + {CSessionConnectionSocket::Accept(343)} + {CSessionConnectionSocket::CheckDataWaiting(663)/W32.258.(The wait operation timed out. (WAIT_TIMEOUT.258))-Timed out}
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstance() - RunClOraAgent() failed.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstance() - End Oracle CrossCheck for jobId=2841371 CommCellId=2 clientId=2506 clientName=Client instanceId=336, instanceName=Instance jobControl=000000B40BEDBB70. CROSSCHECK_STATUS=FAILURE
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - CrossCheckForOraClientAndInstance() failed for clientId=2506 instanceId=336.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Mark failure for instanceId=336 oraSID=Instance.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Mark failure for client[Client] with clientId=2506.
8728  70c0  10/19 20:19:39 2841371 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - Exit for jobId=2841371 CommCellId=2 jobControl=000000B40BEDBB70.
8728  70c0  10/19 20:19:39 2841371 APOracleClient::deleteAF() - Some instances of client [2506] will be skipped due to error.
8728  70c0  10/19 20:19:39 2841371 APOracleInst::deleteAF() - Skip instance [336] on client [2506]
8728  70c0  10/19 20:19:39 2841371 APOracleClient::deleteAF() - Client [2506]: Failed for some or all archive files
8728  70c0  10/19 20:19:39 2841371 APOracleCrossCheck::deleteAF() - Failed for some or all archive files
8728  70c0  10/19 20:19:39 2841371 DATA AGING() - JOB [2841371]: Failed to Cross-Check Oracle Clients. 
8728  70c0  10/19 20:19:39 2841371 APSAPCrossCheck::removeDanglingAF() - Archive files from deleted clients and instances are deleted successfully
8728  70c0  10/19 20:19:39 2841371 ArchPruneBase::getSAPCrossCheckInfo() - Total [0] SAP instances to be cross checked

The Oracle Cross Check is there to ensure that what we have in the Oracle job history matches what Oracle is aware of.  This way, if Oracle expects a certain log file to exist, we don’t delete it.

If the cross check fails, that’s almost always on the Oracle side of things and you should check with your DBA to find out why.

You CAN disable the cross check, but you might delete a log file job that Oracle expects to have access to….so it shouldn’t be done without your DBA being consulted.

Hi @Mike Struening ,

I have requested our Oracle DBA  investigate the crosscheck failures on the DB and The archive logs crossscheck had  no errors.

  This is the error i am getting ”Establishing connection to Client services failed during Crosscheck. Make sure this product's Services are running on the client [IP*server*8400*8402] and check network connectivity between client and CommServe.”

Any tips on how i should go about in solving this issue.

@Makadi Happines 

Based upon the log cuts that you provided is the specific oracle instance still running on the client. Also you can check the ClOraAgent log on the client. As this is the log that the data aging process will use to process the cross check. 

@Mike Struening ,

Checking data aging logs, RunClOraAgent() fails. See logs below

How do I check ClOraAgent logs on a client?

16012 1c54  11/10 18:04:17 2877733 ::RunClOraAgent() - dataAgentAttach-> HostName=1server start...
16012 1c54  11/10 18:06:17 2877733 ::RunClOraAgent() - Attach_RemoteCommand (ClOraAgent) failed:0x80070102:{CCvNetwork::Attach_RemoteCommand(2137)} + {CCvNetwork::Attach_RemoteCommand(2095)} + {CCvNetwork::Accept(1235)} + {CSessionConnectionSocket::Accept(343)} + {CSessionConnectionSocket::CheckDataWaiting(663)/W32.258.(The wait operation timed out. (WAIT_TIMEOUT.258))-Timed out}
16012 1c54  11/10 18:06:17 2877733 OraCrossCheck::CrossCheckForOraClientAndInstance() - RunClOraAgent() failed.
16012 1c54  11/10 18:06:17 2877733 OraCrossCheck::CrossCheckForOraClientAndInstance() - End Oracle CrossCheck for jobId=2877733 CommCellId=2 clientId=2478 clientName=server instanceId=303, instanceName=CC1 jobControl=00000068828EBFE0. CROSSCHECK_STATUS=FAILURE
16012 1c54  11/10 18:06:17 2877733 OraCrossCheck::CrossCheckForOraClientAndInstanceLists() - CrossCheckForOraClientAndInstance() failed for clientId=2478 instanceId=303.

@Makadi Happines 

You will need to get a remote session to the client and in the commvault logs files directory you will find the file ClOraAgent log. Or you can right click on the client view - log files and select the log file from the list.

Hi, How i can run crosscheck and delete backup in comvault?

thanks

@Debora , this happens when you run Data Aging.