DDB High Q&I Time with high Number of Pending Delete Records
Hi,
I have a DDB that has high Q&I times. This is hosted on a an AWS EC2 instance with the disk type being “gp3” with 300 IOPS.
Below is the Databases section.
The Files portion of the DDB is not reporting the same issue.
My concerns also includes the very high Pending Delete Records in it.
I can see that the Data Aging jobs do run as the Prunable Records does go down to zero.
I attempted a manual space reclamation on this AWS S3 library that completes with errors.
When looking through the logs it appears that the Media Agent does not have the required access on the bucket but the access does seem in place when using the Cloud Test too.
Error on server.
Could this be contributing to the high Q&I values?
Any ideas that could assist with this resolving this?
Thanks.
Ignes
Page 1 / 1
@Iggy - DDB Q&I time will be affected due to the slow disk response time and IOPS.
Even for the medium environment, we recommend having 5k dedicated random IOPS for the DDB disk.
IOPS Requirement
The minimum IOPS required for DDB disk for extra large, large and medium MediaAgent is:
You need to check the SIDBPrune.log from the MA to check why the deletion is not happening.
Moving the DDB to the SSD/premium disk will help in improving the Q & I time.
Kindly let me know if you have any queries.
Hi @Arunkumar P,
I see that the disk is not maxed out in terms of IOPS and the other metrics also look sufficient but I will request the volume to be modified to cater for the minimum of 5K IOPS.
Below if during the backup window.
Below is an extract from SIDBPrune.log on the MA.
Multiple “403 Forbidden” errors. I am not sure why because as shown previously the Access check is successful.
I need to mention that the bucket is made up of a Combined Storage Tier.
S3 Standard-IA
S3 Glacier Instant Retrieval
Would this change the behavior of the deletes?
3870 179e 10/25 15:28:45 ### 41-3 OpenIdx:3395 Cannot open Sfile Idx lomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289397/CHUNK_9340410/SFILE_CONTAINER.idx], SysErr S0], Err ]0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}] 3870 1598 10/25 15:28:45 ### CIndexFile::Open:1545 Cannot open index file xomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.} 3870 1598 10/25 15:28:45 ### Failed to open file nomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx]. Line .1546], Error 0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}] 3870 1598 10/25 15:28:45 ### 41-3 OpenIdx:3395 Cannot open Sfile Idx lomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx], SysErr S0], Err ]0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}] 3870 13cb 10/25 15:28:46 ### CIndexFile::Open:1545 Cannot open index file xomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.} 3870 13cb 10/25 15:28:46 ### Failed to open file nomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx]. Line .1546], Error 0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}] 3870 13cb 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx lomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx], SysErr S0], Err ]0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}] 3870 1536 10/25 15:28:46 ### CIndexFile::Open:1545 Cannot open index file xomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.} 3870 1536 10/25 15:28:46 ### Failed to open file nomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx]. Line .1546], Error 0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}] 3870 1536 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx lomi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], SysErr S0], Err ]0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
Cheers.
Ignes
Hi.
I am able to list the SFILES when using “aws s3 ls”
Would the permission be related to delete object?
Cheers.
Iggy
@iggy - We are unable to open/delete the index file and it gives forbidden error.
