Solved

High read IOPS Oracle backup

  • 15 April 2021
  • 5 replies
  • 1202 views

Userlevel 2
Badge +3

hi guys, we have an issue with one of our Oracle VMs. The VM is not really big, the backup data is 1-2 gb in size (size of application). Compression rate is ~70%.

96% of the time the backup is in read phase. The backup take around 30 minutes. The disk is really busy inside the VM. When the backup starts we see 20-30k read IOPS and 100% busy state, which I can’t explain. We can see it on full backups and on incremental backups of the default subclient.

We already limited the read throughput with the RATE setting inside CV to 250MB/s. (before we saw 800-900 MB/s read). Anyway, this do not help with the high read IOPS at the start of the backup. We see this high IOPS for 30-60 seconds and then it falls to 500-1000 during the rest of the backup.

Anything we can do to limit the READ IOPS in this case? What is the reason for such high read iops for this small database? What exactly is happening at the start of the rman backup? On the backend storage I can not see any bottleneck. We use All Flash (SSD) Storage (FC) for all VMs.

No difference when we disable compression or use our Media Agents for compression.

This is the rman script:

Rman Script:
[CONFIGURE CONTROLFILE AUTOBACKUP ON;
run {
allocate channel ch1 type 'sbt_tape'
RATE 250M PARMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so, BLKSIZE=1048576 ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1,ThreadCommandLine= -cn vm1 -vm Instance001)"
TRACE 0;
send "BACKUP -jm 32813 -a 2:2998 -cl 2128 -ins 482 -at 22 -j 3476065 -jt 3476065:4:1:0:48491 -bal 0 -t 2 -ms 1 -data";
setlimit channel ch1 maxopenfiles 8;
backup
incremental level = 1
filesperset = 8
format='3476065_%d_%U'
database
include current controlfile ;
}
exit;
]

We are on 11.20.32

icon

Best answer by Edd Rimmer 16 April 2021, 08:55

View original

5 replies

Userlevel 4
Badge +7

Hello,

Can you confirm if you have block change tracking enabled for the database? This behavior sounds like block change tracking may not be enabled - if it is not, Oracle will scan all the blocks on an incremental backup for changes, instead of using a file to track changes.

You didn’t let us know the Oracle version but here’s doc. ref from version 12 → https://docs.oracle.com/database/121/ADMQS/GUID-3BAA0D48-CA35-4CD7-810E-50C703DC6FEB.htm

Also if you have not already done so, would suggest reading the information @ https://documentation.commvault.com/commvault/v11/article?p=20285.htm

Let me know what you discover :grinning:

Regards

Userlevel 2
Badge +3

Hey Edd,

thanks for the info regarding CBT. Of course it was disabled. After we enabled CBT the backup time droped hard. Still we can see high read iops.. but we need to investigate this further and check if there is “a real impact” on production after CBT is activ.

Thanks and I’ll update as soon as I know more. :-)

Userlevel 7
Badge +23

@Dimitri , following up to see how @Edd Rimmer ‘s suggestions worked out.

Thanks!

Userlevel 2
Badge +3

At the moment we see less issues, but there is still an impact. We can close here, as this helped for sure but we need more time to investigate this further.

Userlevel 7
Badge +23

Thanks, @Dimitri !  If you respond here, I can ‘unmark’ the best answer.

Reply