DDB High Q&I Time with high Number of Pending Delete Records

@Iggy - DDB Q&I time will be affected due to the slow disk response time and IOPS. 

Even for the medium environment, we recommend having 5k dedicated random IOPS for the DDB disk. 

IOPS Requirement

The minimum IOPS required for DDB disk for extra large, large and medium MediaAgent is:

300 IOPS is way lesser which would affect the DDB performance and will result in high Q & I. 

https://documentation.commvault.com/2023e/expert/8825_testing_iops_of_deduplication_database_disk_on_windows.html

You need to check the SIDBPrune.log from the MA to check why the deletion is not happening. 

Moving the DDB to the SSD/premium disk will help in improving the Q & I time. 

Kindly let me know if you have any queries. 

Hi @Arunkumar P,

I see that the disk is not maxed out in terms of IOPS and the other metrics also look sufficient but I will request the volume to be modified to cater for the minimum of 5K IOPS.

Below if during the backup window.

Below is an extract from SIDBPrune.log on the MA.

Multiple “403 Forbidden” errors. I am not sure why because as shown previously the Access check is successful.

I need to mention that the bucket is made up of a Combined Storage Tier.

S3 Standard-IA

S3 Glacier Instant Retrieval

Would this change the behavior of the deletes?

3870 179e 10/25 15:28:45 ### 41-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289397/CHUNK_9340410/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1598 10/25 15:28:45 ### CIndexFile::Open:1545 Cannot open index file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}
3870 1598 10/25 15:28:45 ### Failed to open file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx]. Line [1546], Error [0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1598 10/25 15:28:45 ### 41-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 13cb 10/25 15:28:46 ### CIndexFile::Open:1545 Cannot open index file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}
3870 13cb 10/25 15:28:46 ### Failed to open file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx]. Line [1546], Error [0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 13cb 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1536 10/25 15:28:46 ### CIndexFile::Open:1545 Cannot open index file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}
3870 1536 10/25 15:28:46 ### Failed to open file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx]. Line [1546], Error [0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1536 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]

Cheers.

Ignes

Hi.

I am able to list the SFILES when using “aws s3 ls”

Would the permission be related to delete object?

Cheers.

Iggy

@iggy - We are unable to open/delete the index file and it gives forbidden error. 

3870 1536 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]

#define E_MM_HTTP_ERROR_FORBIDDEN                               44106

This http error returned by the S3 and it is worth involving the cloud vendor to troubleshoot further on this part.