Does commvault make use of Multi-part upload to S3 compatible HCP/HCP-CS storage? If yes, what's the part size?

Hello @Jass Gill 

I have asked our internal team around these questions for the size and if it can be modified but i do know we take advantage of Multi-Part Upload. 

I will keep you posted on Dev’s response around the size and if it can be changed. 

Can you advise what version you are running and why you want to know this info? Is there an issue you are having and you are trying to fix? 

Kind regards

Albert Williams

I do not suspect Commvault is actually using MPU as there is no need for it. The object size that Commvault writes to a cloud library is 8MB for operations with deduplication enabled and 32MB for non-deduplicated. In general the advise is to use MPU starting with object sizes larger than 100MB. There is no difference in case WORM is enabled/disabled. Only difference you will notice in when you use deduplication. 

In regards to log insights see: https://documentation.commvault.com/2023e/essential/monitoring_and_troubleshooting_faster_restores_from_amazon_s3_glacier.html#commvault-log-files 

Hey Albert,

Thanks for looking into this.
 

So basically we are doing some internal qualification for a new release of an object storage system and observed that when I use WORM-enabled storage policy to backup a file. I can see from the Object store logs that the PUT requests include partNumber and uploadID. See the example below.

PUT /2.6wormtest/Z5YO4V_09.06.2024_10.43/CV_MAGNETIC/V_342/CHUNK_1294/SFILE_CONTAINER_020?partNumber=1&uploadId=7ffffe6e38ff88e4 HTTP/1.1

When using a NONWORM policy to backup the same file, the received PUT request does not include the partNumber or uploadID. See the example below.

PUT /windowsnonworm/1DWYIV_09.11.2024_10.49/CV_MAGNETIC/V_8220/CHUNK_84999/SFILE_CONTAINER_001.FOLDER/0 HTTP/1.1" 200

My question is, does Commvault not employ multipart upload during a backup with the standard (WORM disabled) policy?

Yes. For Storage Policies that does not support Micro Pruning (WORM and Archive Storages), we use multi-part uploads. For Standard Dedup Storage Policies (without WORM or Archive Storages), we don't use multipart upload.

@Deepthi thanks for sharing! I learned something new ;-) Is my feedback in regards to object sizes correct?