S3-IA early deletion

  • 15 February 2022
  • 10 replies
  • 692 views

Userlevel 1
Badge +3

I am doing this sortof.  I have an S3-IA bucket and send my Aux copies to it.  I am just not doing the tiering to Glacier or something else.  Everything stays in S3IA.  My Aux copy retention is for 91 days, but I am getting blasted with AWS charges for early deletes.  Does anyone know how I can find what is deleting early?  CV support verified I have proper retention set.

Thanks,

Stephanie


If you have a question or comment, please create a topic

10 replies

Userlevel 7
Badge +23

Hey @SysadminStephanie - I created a new thread from your reply as its an interesting topic.

 

S3-IA always bills in 30 day increments, so an object lasting less than 30 days is billed for 30 days regardless. I missed your retention initially but I now see its 91 days. 

You may be able to monitor deletes in sidbphysicaldeletes.log for the MA’s that handle that library - but you’d need to know a few details to match up the right copy etc. Let me think about this one and see if others have any ideas.

Userlevel 7
Badge +23

@SysadminStephanie , there is a way to see the chunks in the database, and determine which jobs belong to them through some SQL querying.

However, in this case you’re almost definitely looking at micro pruning.

For full context (for anyone else following along), we have to talk about how deduplication works when writing, as well as pruning.

Job 1 contains blocks A, B and C.  The backup process checks the dedupe database (DDB) and sees that these do not belong to any job yet, nor do they exist.  It then writes these 3 blocks to the library, adds all 3 to the Primary table, and adds 1 reference for each block (for this 1 job) to the secondary table for blocks A, B, and C.

Job 2 comes along and contains blocks B, C, and D.  The backup process checks the dedupe database (DDB) and sees that B and C exist, but D does not.  It then writes block D blocks to the library, adds all D to the Primary table, and adds 1 reference for each block (for this 1 job) to the secondary table for blocks B, C, and D.

We now have blocks A, B, C, and D with 2, 2, 2, and 1 reference respectively.

Some time later, based on your retention settings, Job 1 ages off.  The CS sends the list of associated volumes (we’ll just say blocks to make the concept easier) to the pruning Media Agent to process.  Job 1 is made up of blocks A, B, and C.  Therefore, the Media Agent decrements each of those by 1.

This leaves block A with no references, and blocks B, C, and D with 1 more.

This then sends block A to the Zero Refs table (the last of the 3 DDB tables).  when the Media Agent is ready to physically remove blocks, it pulls the list from Zero Ref and makes physical deletions.

So how does this apply to you and the cloud library?  You are likely pruning of small parts of the jobs as time goes by.

However, if you are saying these files are being deleted WITHIN 30 days, then that is something else, potentially.

Once the files are deleted, it’s tough for us to figure out who they DID belong to, since the job is long aged off logically, and the references are long gone.

I would suggest opening a support case to see if they can use an old Commserve database to find the jobs connected to those volumes and backtrack what happened.  Should definitely be possible, though will be tough to figure out on this forum.

Can you share the case number here once you create the incident?

Userlevel 1
Badge +3

My retention is 93 days, in an effort to let commvault age a job before AWS.  However, I am not sure AWS just goes around willy-nilly deleting my ‘files’ aka dedup blocks when they are 90 days old.  Anyway, when job 1 ages at 93 days and decrements the reference counts on blocks A, B, C, they are left with 0 count and can be deleted.  But, they should be 93 days old at the minimum, so I should not get dinged with an AWS early delete there.

I am wondering if the metadata about my deduplication and the reference counts is what AWS is ‘deleting’ by changing it.  AWS does say deleting or updating is an ‘early delete’

 

Incident 220211-370

Userlevel 7
Badge +23

Agreed, the more I think about it, the more I don’t see how any file under 93 days should be deleted at all, save killed jobs, maybe?

I’ve very curious to see what the answer is on this one.

Thank you VERY much for bringing it to the community to discuss!

I’ll monitor the case you provided closely :nerd:

Userlevel 7
Badge +23

First of all, thank you for that tidbit of billing info!  AWS advertises that S3IA charges for early deletes, early deletes being <90 days.  Their billing report actually describes it as ‘deleted or overwritten before 30 days’ 

Yeah it should not be 90 days for S3IA. I can only find references to > 30 days.

 

Source

S3 Standard-IA, and S3 One Zone-IA storage are charged for a minimum storage duration of 30 days, and objects deleted before 30 days incur a pro-rated charge equal to the storage charge for the remaining days. Objects that are deleted, overwritten, or transitioned to a different storage class before 30 days will incur the normal storage usage charge plus a pro-rated charge for the remainder of the 30-day minimum. This includes objects that are deleted as a result of file operations performed by File Gateway. Objects stored for 30 days or longer will not incur a 30-day minimum charge. 

 

@SysadminStephanie - do you know if you run space reclamation at all? Its an option on DDB job verification - might be worth checking to see if any of those types of administrative jobs have been run. That's the only thing I can think of when it comes to a cloud library that matches the pattern you are seeing.

Userlevel 7
Badge +23

If it were killed jobs, I don’t think they would have been aux copied.  And even if they were, that wouldn’t be $600/month.   

I just read through your case….when I saw the cost my jaw dropped.

Hopefully support can get it figured out FAST!

Userlevel 1
Badge +3

First of all, thank you for that tidbit of billing info!  AWS advertises that S3IA charges for early deletes, early deletes being <90 days.  Their billing report actually describes it as ‘deleted or overwritten before 30 days’  Since I am using deduplication, Commvault should not be re-sending or re-writing any chunks.

 

Looking at sidbPhysicalDeletes.log, I get information like this:

  26b0 02/16 06:43:31 Deletion      MountPathID=509 v_1426054\chunk_18747151         Chunk-metadata-file
  24cc 02/16 06:43:31 Deletion      MountPathID=516 v_1426016\chunk_18747269         Chunk-metadata-file
  26b0 02/16 06:43:31 Deletion      MountPathID=509 v_1426054\chunk_18747202         Chunk-metadata-file
  26b0 02/16 06:43:32 Deletion      MountPathID=509 v_1426054\chunk_18747242         Chunk-metadata-file
 

Does MountPathID=xxx match up with anything I can see, & filter on, from Commvault CommCell Console or from the AWS bucket?

Userlevel 7
Badge +23

@SysadminStephanie , you should be able to browse the cloud library via the Cloud Storage Explorer Tool:

https://documentation.commvault.com/11.25/expert/9225_cloud_storage_explorer_tool.html

Userlevel 1
Badge +3

Is there any tool to find the retention of blocks or chunks?  I have already checked my storage policy copy for job status, including retention.  

Is it possible that the deduplication process is updating something in AWS bucket that is being counted as a delete?

Userlevel 1
Badge +3

If it were killed jobs, I don’t think they would have been aux copied.  And even if they were, that wouldn’t be $600/month.