Solved

Issue with Oracle Backups

2 years ago
20 July 2022
10 replies
2002 views

Hello.

I'm facing a critical weird issue with oracle RACs backup.

we have 24 RAC, we do a daily full backup without any issues.

we decided to change the backup strategy due to the massive growth of the databases, so we changed the schedules to be weekly full, daily incremental (cumulative) and hourly logs.

once i changed the schedule policy some of the racs incremental backups was failing due to the below error

Error Code: w19:599]
Description: Loss of control process ClOraAgent. Possible causes: 1. The control process has unexpectedly died. Check crash dump or core file. 2. The communication to the control process machine might have gone down due to network errors. 3. If the machine is a cluster, it may have failed over. 4. The machine may have rebooted. 5. Antivirus/Malware software may be blocking the process. Please ensure that all exclusions are in place: http://documentation.commvault.com/commvault/v11/article?p=8665.htm

also, the below event is in the events of the job.

DataServer-IP on -site-a-ma-*Site-A-MA*8400*0] is not ready

i have checked the network connection between the mediaagent and the oracle servers

checked the readiness

checked the status of the instance

all the above was ready with no issues

when i try to preview script for the job i get this message after long wait.

and the backup start with a discover phase which stay for a long time then goes to pending status

i have attached the ora log

appreciate your fast assistance.

Page 1 / 1

@Muhammad Abdullah - Based on the error messages in the log it’s indicating that there is a user permission issue. Recommend performing test connection as user and validate proper user permission are in place.

### Permissions Error ###

140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - Unable to connect as sysdba output = p
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 20 03:50:59 2022
Version 19.12.0.0.0

SQL> SQL> SQL> SQL> ERROR:
ORA-01031: insufficient privileges

Hello @NVFD411

permission is in place and i have tested it , and you can see in the same log that it connected later to the db

140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - Unable to connect as sysdba output =
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 20 03:50:59 2022
Version 19.12.0.0.0

SQL> SQL> SQL> SQL> ERROR:
ORA-01031: insufficient privileges

SQL> SQL> SQL> SQL> SQL> USER is ""
SQL> ]
140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - alternating to sysbackup
140119 22357 07/20 03:50:59 5980422 ::GetNlsLang() - fetching registry key NLS_LANG
140119 22357 07/20 03:50:59 5980422 ::GetNlsLang() - registry key NLS_LANG not set therefore setting NLS_LANG to American_America.US7ASCII
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/sqlplus
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
140119 22357 07/20 03:50:59 5980422 OraChildProcess::InitializeUnix() - Parent: child process pid= 140271
140119 22357 07/20 03:50:59 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:50:59 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - Successfully Connecting to Oracle as sysbackup
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/rman
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
140119 22357 07/20 03:50:59 5980422 OraChildProcess::InitializeUnix() - Parent: child process pid= 140279
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Finalize() - start
140119 22357 07/20 03:51:02 5980422 ClOraAgent::CheckOraSBTStatus(15247) - Running DB Mode check
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - locking iOraBrowseDB_892]
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - locked kOraBrowseDB_892]
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - received oraObjNameList = sB O]
140119 22357 07/20 03:51:02 5980422 OraAgentBase20::BrowseInit(5741) - Initialize0(oraUseraoracle] oraHome=[/oracle/base/home] oraSID=adwhdb1])
140119 22357 07/20 03:51:02 5980422 ::GetNlsLang() - fetching registry key NLS_LANG
140119 22357 07/20 03:51:02 5980422 ::GetNlsLang() - registry key NLS_LANG not set therefore setting NLS_LANG to American_America.US7ASCII
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - GetInfo ...
140119 22357 07/20 03:51:02 5980422 OraObject::GetOraMode() - strictSID = 0
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/sqlplus
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
140119 22357 07/20 03:51:02 5980422 OraChildProcess::InitializeUnix() - Parent: child process pid= 140419
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Finalize() - start
140119 22357 07/20 03:51:02 5980422 OraObject::GetOraMode() - oraMode = READ WRITE.
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/sqlplus
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle

Does the user have sysdba privilege? You can check by running the following:

select * from v$pwfile_users;

Dear @NVFD411

the user has sysbackup not sysdba according to commvault sysbackup is sufficient

An Oracle user account with the SYSDBA or SYSBACKUP (in Oracle version 12 or higher) privilege can access the Oracle target database.

https://documentation.commvault.com/11.24/essential/104521_oracle_user_accounts.html

and the backup was working fine with these previliges for a long time, this is not a new environment.

@Muhammad Abdullah - Understood. We can continue to run through the troubleshooting process here or open a case (probably easiest - since we can review live and gather all logs needed)?

If you would like to proceed here we need the output for the following:

Share screenshot or details of the oracle instance properties
- Have DBA run #sqlplus <test connection string based on user details in oracle instance properties> i.e. sqlplus userid/pwd@string as <sysdba/sysbackup>
- Provide above results
Provide ClOraAgent.log(s)

Hello @NVFD411

i have already opend a case.

i have just finished a remote session with one of the engineers and he said that in discover phase the client is trying to communicate with the mediaagent and access the mounthpath but it cant, and it should not do this in discover phase, it should communicate with the commserv only.

625837 98cad 07/20 16:05:41 5987074 AbstractMountingPointDriver::InitializeDataServerAPI(53) - Initializing DataServerAPI with subclientID e9635], server name site-a-ma-03.efinance.com.eg*Site-A-MA-03*8400*0], path J:\BH1D2E_05.30.2022_09.14\CV_MAGNETIC] prefixInstanceGuid etrue]
609819 94e1b 07/20 16:06:08 5987074 AbstractMountingPointDriver::CreateRootFolder(427) - DataServerAPI checking root folder oE:\H9VJSY_05.12.2021_13.50\CV_MAGNETIC/CV_APP_DUMPS/908AF983-75B1-42AD-BC61-CE76D0552D1E/1658324397] appendPath[/1658324397]
609819 94e1b 07/20 16:06:08 5987074 getMediaAgentCSHostName() - Invalid platform type.
609819 94e1b 07/20 16:06:08 5987074 AbstractMountingPointDriver::CreateRootFolder(432) - Failed to call Exists, DataServerIP is not ready error code = o-335360743] errorString=t0xEC02CD19:{CDataServerFS::Exists(4286)} + {DNFS::NfsFile::ConvertToMMError(2691)/MM.52505-0xEC02CD19:{DNFS::CNfsSessionPool::addNfsFiletoSession(285)} + {DNFS::CNfsSession::initiateSession(997)} + {DNFS::CNfsSession::doNfsMount(961)/MM.52505-Failed to mount Site-A-MA-03:E:/H9VJSY_05.12.2021_13.50/CV_MAGNETIC/CV_APP_DUMPS/908AF983-75B1-42AD-BC61-CE76D0552D1E for sessione4], fd:7,port:26838 error- rpc_service: socket error Connection timed out(110).}}]

regarding the troubleshooting, im not really an oracle guy, but you can explain to me what do check and i can do it and send a screenshots

Please share your case number so we can track.

Hello @NVFD411

thank you for the support.

the case was solved.

root cause: disk cache was enabled on archive log schedule policy.

disk cache requires a dataserver configuration and requires specific ports to be enabled.

i have disabled the disk cache option and the RACs started to comeback successively.

please mark this topic as solved.

thanks for you inputs /efforts.

Glad to hear it!

Hello @NVFD411

thank you for the support.

the case was solved.

root cause: disk cache was enabled on archive log schedule policy.

disk cache requires a dataserver configuration and requires specific ports to be enabled.

i have disabled the disk cache option and the RACs started to comeback successively.

please mark this topic as solved.

thanks for you inputs /efforts.

Thanks for raising this peculiar issue in the blog. This has helped me in resolving the Oracle Archive Log backup issue.

Reply

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded