Skip to main content
Solved

Issue with Oracle Backups

 Hello.

 

I'm facing a critical weird issue with oracle RACs backup.

we have 24 RAC, we do a daily full backup without any issues. 

we decided to change the backup strategy due to the massive growth of the databases, so we changed the schedules to be weekly full, daily incremental (cumulative)  and hourly logs.

once i changed the schedule policy some of the racs incremental backups was failing due to the below error 

 

Error Code: w19:599]
Description: Loss of control process ClOraAgent. Possible causes: 1. The control process has unexpectedly died. Check crash dump or core file. 2. The communication to the control process machine might have gone down due to network errors. 3. If the machine is a cluster, it may have failed over. 4. The machine may have rebooted. 5. Antivirus/Malware software may be blocking the process. Please ensure that all exclusions are in place: http://documentation.commvault.com/commvault/v11/article?p=8665.htm

also, the below event is in the events of the job.

DataServer-IP on -site-a-ma-*Site-A-MA*8400*0] is not ready

 

i have checked the network connection between the mediaagent and the oracle servers

checked the readiness

checked the status of the instance 

all the above was ready with no issues 

when i try to preview script for the job i get this message after long wait.

and the backup start with a discover phase which stay for a long time then goes to pending status 

 i have attached the ora log 

 

appreciate your fast assistance.

 

 

10 replies

N
Rank
Userlevel 2
Badge +7

@Muhammad Abdullah - Based on the error messages in the log it’s indicating that there is a user permission issue.  Recommend performing test connection as user and validate proper user permission are in place. 

### Permissions Error ###

140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - Unable to connect as sysdba output = [
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 20 03:50:59 2022
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

SQL> SQL> SQL> SQL> ERROR:
ORA-01031: insufficient privileges

 

Mark Cerda
M
Rank
Userlevel 2
Badge +9

Hello @NVFD411 

permission is in place and i have tested it , and you can see in the same log that it connected later to the db 

 

140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - Unable to connect as sysdba output = [
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 20 03:50:59 2022
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

SQL> SQL> SQL> SQL> ERROR:
ORA-01031: insufficient privileges


SQL> SQL> SQL> SQL> SQL> USER is ""
SQL> ]
140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - alternating to sysbackup
140119 22357 07/20 03:50:59 5980422 ::GetNlsLang() - fetching registry key NLS_LANG
140119 22357 07/20 03:50:59 5980422 ::GetNlsLang() - registry key NLS_LANG not set therefore setting NLS_LANG to American_America.US7ASCII
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/sqlplus
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
140119 22357 07/20 03:50:59 5980422 OraChildProcess::InitializeUnix() - Parent: child process pid= 140271
140119 22357 07/20 03:50:59 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:50:59 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:50:59 5980422 ClOraAgent::GetBackupUserRole() - Successfully Connecting to Oracle as sysbackup
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/rman
140119 22357 07/20 03:50:59 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
140119 22357 07/20 03:50:59 5980422 OraChildProcess::InitializeUnix() - Parent: child process pid= 140279
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Finalize() - start
140119 22357 07/20 03:51:02 5980422 ClOraAgent::CheckOraSBTStatus(15247) - Running DB Mode check
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - locking [OraBrowseDB_892]
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - locked [OraBrowseDB_892]
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() -  received oraObjNameList = [B O]
140119 22357 07/20 03:51:02 5980422 OraAgentBase20::BrowseInit(5741) - Initialize0(oraUser[oracle] oraHome=[/oracle/base/home] oraSID=[dwhdb1])
140119 22357 07/20 03:51:02 5980422 ::GetNlsLang() - fetching registry key NLS_LANG
140119 22357 07/20 03:51:02 5980422 ::GetNlsLang() - registry key NLS_LANG not set therefore setting NLS_LANG to American_America.US7ASCII
140119 22357 07/20 03:51:02 5980422 OraBaseBrowser::BrowseDB() - GetInfo ...
140119 22357 07/20 03:51:02 5980422 OraObject::GetOraMode() - strictSID = 0
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/sqlplus
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
140119 22357 07/20 03:51:02 5980422 OraChildProcess::InitializeUnix() - Parent: child process pid= 140419
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Wait() - Child OK
140119 22357 07/20 03:51:02 5980422 OraChildProcess::Finalize() - start
140119 22357 07/20 03:51:02 5980422 OraObject::GetOraMode() - oraMode = READ WRITE.
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent path = /oracle/base/home/bin/sqlplus
140119 22357 07/20 03:51:02 5980422 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
 

N
Rank
Userlevel 2
Badge +7

Does the user have sysdba privilege? You can check by running the following:

select * from v$pwfile_users;

Mark Cerda
M
Rank
Userlevel 2
Badge +9

Dear @NVFD411 

 

the user has sysbackup not sysdba  according to commvault sysbackup is sufficient 

 

An Oracle user account with the SYSDBA or SYSBACKUP (in Oracle version 12 or higher) privilege can access the Oracle target database.

https://documentation.commvault.com/11.24/essential/104521_oracle_user_accounts.html

 

and the backup was working fine with these previliges for a long time, this is not a new environment.

 

N
Rank
Userlevel 2
Badge +7

@Muhammad Abdullah - Understood.  We can continue to run through the troubleshooting process here or open a case (probably easiest - since we can review live and gather all logs needed)?  

If you would like to proceed here we need the output for the following:

  • Share screenshot or details of the oracle instance properties
    • Have DBA run #sqlplus <test connection string based on user details in oracle instance properties> i.e. sqlplus userid/pwd@string as <sysdba/sysbackup> 
    • Provide above results
  • Provide ClOraAgent.log(s)
Mark Cerda
M
Rank
Userlevel 2
Badge +9

Hello @NVFD411 

 

i have already opend a case. 

i have just finished a remote session with one of the engineers and he said that in discover phase the client is trying to communicate with the mediaagent and access the mounthpath but it cant, and it should not do this in discover phase, it should communicate with the commserv only. 

 

625837 98cad 07/20 16:05:41 5987074 AbstractMountingPointDriver::InitializeDataServerAPI(53) - Initializing DataServerAPI with subclientID [9635], server name [site-a-ma-03.efinance.com.eg*Site-A-MA-03*8400*0], path [J:\BH1D2E_05.30.2022_09.14\CV_MAGNETIC] prefixInstanceGuid [true]
609819 94e1b 07/20 16:06:08 5987074 AbstractMountingPointDriver::CreateRootFolder(427) - DataServerAPI checking root folder [E:\H9VJSY_05.12.2021_13.50\CV_MAGNETIC/CV_APP_DUMPS/908AF983-75B1-42AD-BC61-CE76D0552D1E/1658324397] appendPath[/1658324397]
609819 94e1b 07/20 16:06:08 5987074 getMediaAgentCSHostName() - Invalid platform type.
609819 94e1b 07/20 16:06:08 5987074 AbstractMountingPointDriver::CreateRootFolder(432) - Failed to call Exists, DataServerIP is not ready error code = [-335360743] errorString=[0xEC02CD19:{CDataServerFS::Exists(4286)} + {DNFS::NfsFile::ConvertToMMError(2691)/MM.52505-0xEC02CD19:{DNFS::CNfsSessionPool::addNfsFiletoSession(285)} + {DNFS::CNfsSession::initiateSession(997)} + {DNFS::CNfsSession::doNfsMount(961)/MM.52505-Failed to mount Site-A-MA-03:E:/H9VJSY_05.12.2021_13.50/CV_MAGNETIC/CV_APP_DUMPS/908AF983-75B1-42AD-BC61-CE76D0552D1E for session[4], fd:7,port:26838 error- rpc_service: socket error Connection timed out(110).}}]
 

 

regarding the troubleshooting, im not really an oracle guy, but you can explain to me what do check and i can do it and send a screenshots 

N
Rank
Userlevel 2
Badge +7

Please share your case number so we can track.

Mark Cerda
M
Rank
Userlevel 2
Badge +9

Hello @NVFD411 

 

thank you for the support.

the case was solved. 

root cause: disk cache was enabled on archive log schedule policy. 

disk cache requires a dataserver configuration and requires specific ports to be enabled. 

i have disabled the disk cache option and the RACs started to comeback successively.

 

please mark this topic as solved.

thanks for you inputs /efforts.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Glad to hear it!

https://www.linkedin.com/in/michael-struening
K
Rank
Badge

Hello @NVFD411 

 

thank you for the support.

the case was solved. 

root cause: disk cache was enabled on archive log schedule policy. 

disk cache requires a dataserver configuration and requires specific ports to be enabled. 

i have disabled the disk cache option and the RACs started to comeback successively.

 

please mark this topic as solved.

thanks for you inputs /efforts.

Thanks for raising this peculiar issue in the blog. This has helped me in resolving the Oracle Archive Log backup issue. 😀 

Reply


Terms & ConditionsCookie settings