Oracle RAC Backup

Hi @gurkanipek 

Could you please elaborate on what happens the server freezes 

Oracle db is inaccessible or the entire server is hung or unable none of the users are able to connect to the server ?Or the application running on the server freezes ?

If you are able to putty into the server check if any of the process is consuming high CPU or memory.

To isolate this further, please run a native RMAN backup and check if you are running into the same issue?

Regards,

Gowri Shankar

Hi @Gowri Shankar

When I start the backup job the application on the db freezes and users can’t connect to the application. After the kill backup job, the problem is solved.

CPU and memory utilization is normal.

I’ve tried the RMAN backup, there is no problem with the RMAN backup.

Thanks,

Hi @gurkanipek 

Could you share the RMAN log from the job that froze the db.

Provide the output of show all from rman interface 

Regards,

Hi @Gowri Shankar 

I will share it tomorrow.

Thanks,

Hi @Gowri Shankar,

Your requested logs are as below.

Thanks,

CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 10 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY;
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+.****/snapcf_***1.f';
 

Rman Script:
[run {
allocate channel ch1 type 'sbt_tape' connect sys/********@xxx
PARMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so, BLKSIZE=1048576 ENV=(CV_mmsApiVsn=2,ThreadCommandLine= -vm Instance001)"
TRACE 0;
allocate channel ch2 type 'sbt_tape' connect sys/********@xxx
PARMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so, BLKSIZE=1048576 ENV=(CV_mmsApiVsn=2,ThreadCommandLine= -vm Instance001)"
TRACE 0;
send "BACKUP -jm 65581 -a 2:397 -cl 188 -ins 77 -at 80 -j 94490 -jt 94490:5:1:0:0:32104 -bal 1 -t 1 -ms 2 -logs  -useCvNwSrv -mhn db2*8400*8402";
setlimit channel ch1 maxopenfiles 8;
setlimit channel ch2 maxopenfiles 8;
sql "alter system archive log current";
backup
filesperset = 32
format='94490_%d_%U'
(archivelog  all   not backed up 2 times );
delete noprompt archivelog  until time = 'sysdate-15'  backed up 1 times to sbt ;
}
exit;
]
Rman Log:[
Recovery Manager: Release 19.0.0.0.0 - Production on Wed Dec 7 11:12:52 2022
Version 19.15.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
RMAN>
connected to target database: xxxx (DBID=1656399764)
using target database control file instead of recovery catalog
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17>
allocated channel: ch1
channel ch1: SID=1626 instance= xxx1 device type=SBT_TAPE
channel ch1: CommVault Systems for Oracle: Version 11.0.0(BUILD80)
allocated channel: ch2
channel ch2: SID=1883 instance= xxx2 device type=SBT_TAPE
channel ch2: CommVault Systems for Oracle: Version 11.0.0(BUILD80)
sent command to channel: ch1
sent command to channel: ch2
sql statement: alter system archive log current
Starting backup at Dec 07 2022 11:13:02
current log archived
channel ch1: starting archived log backup set
channel ch1: specifying archived log(s) in backup set
input archived log thread=1 sequence=5710 RECID=16180 STAMP=1122474683
input archived log thread=2 sequence=6571 RECID=16181 STAMP=1122476482
input archived log thread=1 sequence=5711 RECID=16184 STAMP=1122476483
input archived log thread=2 sequence=6572 RECID=16188 STAMP=1122477896
input archived log thread=1 sequence=5712 RECID=16186 STAMP=1122477709
input archived log thread=1 sequence=5713 RECID=16190 STAMP=1122479512
input archived log thread=2 sequence=6573 RECID=16192 STAMP=1122479685
input archived log thread=1 sequence=5714 RECID=16194 STAMP=1122481307
input archived log thread=2 sequence=6574 RECID=16196 STAMP=1122481484
input archived log thread=1 sequence=5715 RECID=16198 STAMP=1122483108
input archived log thread=2 sequence=6575 RECID=16200 STAMP=1122483286
input archived log thread=1 sequence=5716 RECID=16202 STAMP=1122484905
input archived log thread=2 sequence=6576 RECID=16204 STAMP=1122485086
input archived log thread=1 sequence=5717 RECID=16208 STAMP=1122486704
input archived log thread=2 sequence=6577 RECID=16206 STAMP=1122486700
input archived log thread=2 sequence=6578 RECID=16210 STAMP=1122487122
input archived log thread=1 sequence=5718 RECID=16212 STAMP=1122488507
input archived log thread=2 sequence=6579 RECID=16214 STAMP=1122488918
channel ch1: starting piece 1 at Dec 07 2022 11:13:16
channel ch2: starting archived log backup set
channel ch2: specifying archived log(s) in backup set
input archived log thread=2 sequence=6499 RECID=15894 STAMP=1122142542
input archived log thread=1 sequence=5638 RECID=15892 STAMP=1122139706
input archived log thread=1 sequence=5639 RECID=15898 STAMP=1122156371
input archived log thread=2 sequence=6500 RECID=15896 STAMP=1122153659
input archived log thread=2 sequence=6501 RECID=15900 STAMP=1122158592
input archived log thread=1 sequence=5640 RECID=15902 STAMP=1122262163
input archived log thread=2 sequence=6502 RECID=15904 STAMP=1122278923
channel ch2: starting piece 1 at Dec 07 2022 11:13:16
channel ch1: finished piece 1 at Dec 07 2022 11:14:21
piece handle=94490_xxx_ag1epcic_1360_1_1 tag=TAG20221207T111304 comment=API Version 2.0,MMS Version 11.0.0.80
channel ch1: backup set complete, elapsed time: 00:01:05
channel ch1: starting archived log backup set
channel ch1: specifying archived log(s) in backup set
input archived log thread=1 sequence=5648 RECID=15936 STAMP=1122320660
input archived log thread=2 sequence=6510 RECID=15934 STAMP=1122318840
input archived log thread=2 sequence=6511 RECID=15940 STAMP=1122351592
input archived log thread=1 sequence=5649 RECID=15937 STAMP=1122329545
input archived log thread=1 sequence=5650 RECID=15942 STAMP=1122365941
input archived log thread=2 sequence=6512 RECID=15944 STAMP=1122368777
input archived log thread=1 sequence=5651 RECID=15945 STAMP=1122369848
channel ch1: starting piece 1 at Dec 07 2022 11:14:21
RMAN-03009: failure of backup command on ch2 channel at 12/07/2022 11:14:21
ORA-19502: write error on file "94490_xxx_ah1epcic_1361_1_1", block number 10029057 (block size=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
sbtwrite2: Job[94490] thread[29648]: Received Stop Thread Command.
channel ch2 disabled, job failed on it will be run on another channel
released channel: ch1
released channel: ch2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch1 channel at 12/07/2022 11:14:46
ORA-19502: write error on file "94490_xxx_ai1epckd_1362_1_1", block number 315393 (block size=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
sbtwrite2: Job[94490] thread[27686]: Received Stop Thread Command.
RMAN>
Recovery Manager complete.
]

Hi @gurkanipek 

Sure this is an Oracle RAC and not an Oracle standalone instance?

Are you using Oracle Standard or Enterprise Edition? Only the EE supports the use of more than one stream / channel.

This error message “ORA-19511: non RMAN, but media manager or vendor specific failure“ is typical for network issue between client and MediaAgents.

Hi @Ralph 

This is an Oracle Enterprise Edition RAC system.
The error message you mentioned occurred after the kill backup job.

I have to kill backup job for resolve the problem.

Thanks,

Hi @gurkanipek 

Thanks.

Are you using FQDNs on clients?
This line indicateds it’s not:
send "BACKUP -jm 65581 -a 2:397 -cl 188 -ins 77 -at 80 -j 94490 -jt 94490:5:1:0:0:32104 -bal 1 -t 1 -ms 2 -logs  -useCvNwSrv -mhn db2*8400*8402";

Should be “-mhn db2FQDN db2*8400*8402”.

Hi @Ralph

It’s okay. I’ve masked the client name.

Thanks,

Hi @gurkanipek 

Thank you for the output and the rman log

RMAN parameters are good. 

If this is a TDE enabled database then follow the recommendations from the link 

https://documentation.commvault.com/2022e/expert/20811_best_practices_for_oracle.html

In case if the issue still persists then please open a support incident and we could review it further.

Thanks 

Gowri Shankar