S3 Standard - Fast enough for a Primary Copy

Hi ​@MountainGoat,

Using Amazon S3 Standard as a primary copy in Commvault is supported and technically feasible, including with deduplication enabled. However, there are important considerations and trade-offs, especially for large databases and restore performance:

Key Points for S3 Standard as Primary Copy

• Supported Use Case:
  Commvault supports using S3 Standard as a primary copy, and deduplication is fully supported on S3 Standard buckets.

• Performance Considerations:

  • Backup:
    Backups to S3 Standard will generally be slower than to on-premises disk due to network latency and cloud API overhead, but S3 Standard is the fastest of the S3 storage classes.
  • Restore:
    Restores from S3 Standard are faster than from S3 Glacier or Archive tiers, but still slower than from local disk. Throughput is highly dependent on:
    • Network bandwidth between the MediaAgent and AWS
    • Number of parallel streams configured
    • MediaAgent and client hardware
    • S3 API rate limits and object size
  • Deduplication:
    Deduplication is supported and can help reduce the amount of data transferred and stored, but restores of deduplicated data from cloud libraries are typically slower than non-deduplicated restores due to the need to rehydrate data from multiple objects and perform random reads across the cloud storage.

• Large Database Restores (e.g., 5TB):

  • Expected Throughput:
    Real-world restore speeds from S3 Standard can vary widely. Typical observed rates (with proper tuning and no network bottlenecks) are in the range of 100–300 MB/s per stream, but this can be lower if there are network or configuration constraints.
  • Restore Time Estimate:
    For a 5TB database, at 200 MB/s, a full restore would take about 7 hours. Actual times may be longer if deduplication is enabled, as random access patterns and cloud API overhead can further reduce throughput.

Regards,

Rohit Ravi

Thanks for the feedback Rohit,
 

That sounds pretty good.

If we were to implement VSA, and Intellisnap, are there any issues with S3 for that purpose?
We will be streaming the “snap”, from the AWS infrastructure, onto the S3 Primary copy.

Hi ​@MountainGoat,

When implementing Virtual Server Agent (VSA) with IntelliSnap and streaming snapshots to an S3 library as the Primary copy, there are several considerations and potential issues to be aware of:

Key Considerations

1. Supported Workflow

   • Commvault supports using IntelliSnap with VSA for AWS EC2 workloads, allowing you to create point-in-time snapshots and then stream the backup data to a secondary copy, such as an S3 library.
   • The backup copy operation reads the snapshots captured by IntelliSnap and streams the data to the S3 library 

2. Performance Bottlenecks

   • Streaming large snapshots to S3 can be impacted by AWS EBS/S3 performance, especially for first-time reads due to AWS "pre-warming" penalties. This can result in slow backup throughput 
   • Multiple pending snapshots on a volume can further reduce performance until all snapshots complete 

3. Proxy and MediaAgent Requirements

   • The VSA proxy used for IntelliSnap backup copy operations must have the MediaAgent package installed to perform mount operations

Regards,

Rohit Ravi