Hi guys :)
Why is the restore performance so bad for big single item files like vmdk’s (100-400GB/hr)?
Got the information that restores (for single Item) only reads from four disks simultaniosly, and the disks are HDDs, so random reads/throughputs are usually slow (by design). Support can’t help because it’s not an issue, and we have to live with it (if we want to stick with HSX). But thats not acceptable by our customers. Is there anything we can do?
Thx
Kr
Single Item Restore Performance
+2
Userlevel 6
+15
- Vaulter
- 326 replies
-
8 February 2024
The performance will be bottlenecked by your hardware. Your hardware is HDDs.. so as you mention reads/throughputs are by design slow.
You can use this tool to validate your hardwares/mount paths performance: https://documentation.commvault.com/2023e/expert/validating_mount_path.html
However, it would appear that support has already analysed the performance and validated it is the hardware? What is the case number?
If your customers aren’t happy with the outcome, have you established what is acceptable to them? How fast should it be? What are their RPOs?
If you’re hardware can’t meet these requirements, either the expectations need to be reset based on your current capability or you need to investigate faster hardware to meet their requirements.
HTH
Regards.
Chris
Userlevel 7
+23
- Vaulter
- 1092 replies
-
8 February 2024
Hi
Missing some context here to provide a solid answer, but in a lot of cases, individual items or granular recoveries are single stream restores. Quite often better performance is gained with multiple streams, not only when reading data, but when writing data to the application as well.
+2
Hi!
During design phase, we got from CommVault the following perfomance sheet:
The two models tested are the following:
• N12 - cluster of three 2U, 12x14 TB LFF HDD servers
• N24 - cluster of three 2U, 24x8 TB LFF HDD servers
Commvault HyperScale X RA Performance
Operation N12 N24 Notes
Baseline backup 11.7 TB/hour 12.5 TB/hour
Baseline backup assumes there is no deduplication in the data stream. The only storage optimization is due to 50% data compression. First full backups are considered baseline.
Subsequent backup 64 TB/hour 100 TB/hour
Assumes 2% daily change rate, daily incremental backup and a weekly full. Most of the performance gain is due to deduplication, eliminating more than 90% of the writes.
Restore 8.2 TB/hour 11 TB/hour
Full restore of the system with overwrite option turned on, ensuring all data gets rewritten back to the source system.
Aux Copy 12 TB/hour 12 TB/hour
Full copy of data from the appliance to an external target.
DASH Copy 26.7 TB/hour 26.7 TB/hour
Full copy of data from the appliance using deduplication.
Synthetic Full 26.1 TB/hour 44 TB/hour
Create a full backup from the latest full backup and subsequent incremental backups.
VSA baseline backup 7.8 TB/hour 7.8 TB/hour
HyperScale nodes as VSA proxies to perform baseline backup of VMware VMs.
VSA subsequent full 18 TB/hour 18 TB/hour
Assumes 2% daily change rate, daily incremental backup and a weekly full. Most of the performance gain is due to deduplication, eliminating more than 90% of writes.
VSA based VM restores 5.2 TB/hour 5.2 TB/hour
Full restore of the VMware VMs
The performance will be bottlenecked by your hardware. Your hardware is HDDs.. so as you mention reads/throughputs are by design slow.
By the design sheet the bottleneck for VM restore shold be aboe 5.2 TB/hr. During the incident we got the information that single stream writes/reads to/from only four HDDs, and not all 24 in the system. But that information was never told to us.
You can use this tool to validate your hardwares/mount paths performance: https://documentation.commvault.com/2023e/expert/validating_mount_path.html
Thanks, we did enough testing
However, it would appear that support has already analysed the performance and validated it is the hardware? What is the case number?
Yes, “Commvault HyperScale™ X Validated Reference Design Performance”. Can you look into that case? Don’t know if i can hand that out because of customer detailed informations - security.
If your customers aren’t happy with the outcome, have you established what is acceptable to them? How fast should it be? What are their RPOs?
RPO is a scheduled backup every 15min, thats not the problem. RTO is (by luck) not yet defined. If we can achieve >=1TB/hr, i guess customer would agree to that.
If you’re hardware can’t meet these requirements, either the expectations need to be reset based on your current capability or you need to investigate faster hardware to meet their requirements.
We can’t change the hardware. It’s already fully implemented, and customer payed for that an expects the given values (Or the 1TB/hr).
HTH
Regards.
Chris
Hi
Missing some context here to provide a solid answer, but in a lot of cases, individual items or granular recoveries are single stream restores. Quite often better performance is gained with multiple streams, not only when reading data, but when writing data to the application as well.
What additional context would you need? Yea, but in our case we need fast single stream restore, as vmdks can’t be backuped/restored with multiple streams
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.