commvault.com commvault.com
Question

DDB High Q&I Time with high Number of Pending Delete Records

  • 25 October 2023
  • 4 replies
  • 155 views
I
Rank
Badge +8

Hi,

I have a DDB that has high Q&I times. This is hosted on a an AWS EC2 instance with the disk type being “gp3” with 300 IOPS.

Below is the Databases section.

 

The Files portion of the DDB is not reporting the same issue.

 

My concerns also includes the very high Pending Delete Records in it.

 

 

I can see that the Data Aging jobs do run as the Prunable Records does go down to zero.

I attempted a manual space reclamation on this AWS S3 library that completes with errors.

When looking through the logs it appears that the Media Agent does not have the required access on the bucket but the access does seem in place when using the Cloud Test too.

 

Error on server.

 

 

Could this be contributing to the high Q&I values?

Any ideas that could assist with this resolving this?

 

Thanks.

Ignes

 

4 replies

Arunkumar P
Rank
Userlevel 2
Badge +7

@Iggy - DDB Q&I time will be affected due to the slow disk response time and IOPS. 

Even for the medium environment, we recommend having 5k dedicated random IOPS for the DDB disk. 

IOPS Requirement

The minimum IOPS required for DDB disk for extra large, large and medium MediaAgent is:

Components

Extra Large

Large

Medium

Suggested IOPS for DDB Disk

20K dedicated Random IOPS

15K dedicated Random IOPS

5K dedicated Random IOPS

 

300 IOPS is way lesser which would affect the DDB performance and will result in high Q & I. 

https://documentation.commvault.com/2023e/expert/8825_testing_iops_of_deduplication_database_disk_on_windows.html

You need to check the SIDBPrune.log from the MA to check why the deletion is not happening. 

Moving the DDB to the SSD/premium disk will help in improving the Q & I time. 

Kindly let me know if you have any queries. 

I
Rank
Badge +8

Hi @Arunkumar P,

I see that the disk is not maxed out in terms of IOPS and the other metrics also look sufficient but I will request the volume to be modified to cater for the minimum of 5K IOPS.

Below if during the backup window.

 

 

Below is an extract from SIDBPrune.log on the MA.

Multiple “403 Forbidden” errors. I am not sure why because as shown previously the Access check is successful.

I need to mention that the bucket is made up of a Combined Storage Tier.

S3 Standard-IA

S3 Glacier Instant Retrieval

Would this change the behavior of the deletes?

 

3870 179e 10/25 15:28:45 ### 41-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289397/CHUNK_9340410/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1598 10/25 15:28:45 ### CIndexFile::Open:1545 Cannot open index file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}
3870 1598 10/25 15:28:45 ### Failed to open file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx]. Line [1546], Error [0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1598 10/25 15:28:45 ### 41-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1289402/CHUNK_9324161/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 13cb 10/25 15:28:46 ### CIndexFile::Open:1545 Cannot open index file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}
3870 13cb 10/25 15:28:46 ### Failed to open file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx]. Line [1546], Error [0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 13cb 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278234/CHUNK_9146229/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1536 10/25 15:28:46 ### CIndexFile::Open:1545 Cannot open index file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], error=0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}
3870 1536 10/25 15:28:46 ### Failed to open file [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx]. Line [1546], Error [0xEC02AC4A:{CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]
3870 1536 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]

 

Cheers.

Ignes

 

 

I
Rank
Badge +8

Hi.

I am able to list the SFILES when using “aws s3 ls”

Would the permission be related to delete object?

 

Cheers.

Iggy

 

Arunkumar P
Rank
Userlevel 2
Badge +7

@iggy - We are unable to open/delete the index file and it gives forbidden error. 

3870 1536 10/25 15:28:46 ### 43-3 OpenIdx:3395 Cannot open Sfile Idx [omi-retail-commvault-nonprod/QDQOT9_09.05.2023_09.16/CV_MAGNETIC/V_1278487/CHUNK_9151170/SFILE_CONTAINER.idx], SysErr [0], Err [0xEC02AC4A:{IndexHdrMagic>::Open(1709)} + {CCloudFile::Open(1210)} + {CVBaseRemoteFile::GetLastMMErrorCode(274)/MM.44106-403 Forbidden.}]

#define E_MM_HTTP_ERROR_FORBIDDEN                               44106

This http error returned by the S3 and it is worth involving the cloud vendor to troubleshoot further on this part. 

 

Reply


Terms & ConditionsCookie settings