hi guys, we have an issue with one of our Oracle VMs. The VM is not really big, the backup data is 1-2 gb in size (size of application). Compression rate is ~70%.
96% of the time the backup is in read phase. The backup take around 30 minutes. The disk is really busy inside the VM. When the backup starts we see 20-30k read IOPS and 100% busy state, which I can’t explain. We can see it on full backups and on incremental backups of the default subclient.
We already limited the read throughput with the RATE setting inside CV to 250MB/s. (before we saw 800-900 MB/s read). Anyway, this do not help with the high read IOPS at the start of the backup. We see this high IOPS for 30-60 seconds and then it falls to 500-1000 during the rest of the backup.
Anything we can do to limit the READ IOPS in this case? What is the reason for such high read iops for this small database? What exactly is happening at the start of the rman backup? On the backend storage I can not see any bottleneck. We use All Flash (SSD) Storage (FC) for all VMs.
No difference when we disable compression or use our Media Agents for compression.
This is the rman script:
[CONFIGURE CONTROLFILE AUTOBACKUP ON;
allocate channel ch1 type 'sbt_tape'
RATE 250M PARMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so, BLKSIZE=1048576 ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1,ThreadCommandLine= -cn vm1 -vm Instance001)"
send "BACKUP -jm 32813 -a 2:2998 -cl 2128 -ins 482 -at 22 -j 3476065 -jt 3476065:4:1:0:48491 -bal 0 -t 2 -ms 1 -data";
setlimit channel ch1 maxopenfiles 8;
incremental level = 1
filesperset = 8
include current controlfile ;
We are on 11.20.32
Best answer by Edd Rimmer
Can you confirm if you have block change tracking enabled for the database? This behavior sounds like block change tracking may not be enabled - if it is not, Oracle will scan all the blocks on an incremental backup for changes, instead of using a file to track changes.
You didn’t let us know the Oracle version but here’s doc. ref from version 12 → https://docs.oracle.com/database/121/ADMQS/GUID-3BAA0D48-CA35-4CD7-810E-50C703DC6FEB.htm
Also if you have not already done so, would suggest reading the information @ https://documentation.commvault.com/commvault/v11/article?p=20285.htm
Let me know what you discover