We had a customer who did not make any changes and all of a sudden their backup size had almost doubled; the backups are configured using Commvault VADP. We host the VM in a shared VMWARE Cluster for them. I think they added only one addtional VM and the backup size increased in all the VMS
The customer was really concerned with this high bill as we bill them on Backup size; I was looking online and found that this could happen due to thick provisioning and disks are not zeroed and running sdelete utility might help.
How is this backup size calculated.
I thought if the customer had thin disk provisioning this backup size would be total of what the customer is utilizing however that does not appear to be the case; one of the VM with VM size of 80GB and used size of 40 GB is still showing a backup size of 80GB; so a bit confused; and wondering if we run sdelete would this be resolved or would this reoccur and what caused this issue. Please see the screenshot below from Aug and Sept 2021
Best answer by Peter KalinicView original
@Theseeker , can you confirm if the actual backups are bigger, or if you are only seeing things increase in the Chargeback report? there’s an older issue that distorted the view of jobs in the chargeback report, though the actual backups were the normal size.
We use the backup size to bill the customer basis the screenshot I had pasted before...the backup for the hot add went to 1tb from 600gb...we use the last synthetic full for the month...
Ok, so the actual size was larger, not just the report.
Your last sentence raised a question….do you see an Incremental suddenly jump in size? how are this month’s Incs compared to last month?
We shouldn’t discount the notion that this could just be a bigger amount of data across the Incs.
okay this is the screenshot with the largest incremental size from backup history: however the wired thing is when I look into the details backup size does not add up to size of application although transfer time show over 392.97 GB of data transfer
another backup set reflects large app size however this at least closely matches the VM backup size as well
This isn’t the first topic we’ve had about vm backup size stats looking off…
I doubt it is a coincidence.
edit: no need to open a case, I have the answer below!
I have more information!
What’s happening here is that we no longer discard deleted files during vm backups.
That’s the high level answer; the deeper answer is a bit more detailed:
When vms delete a file, they dirty the disk (i.e. it’s not blank, but there’s no file there).
When we check with CBT, we are told those blocks have changed (because they have) even though no file is there.
To cut down on space, we were looking at the changed list, then seeing what files were actually active, and discarding the ‘deleted’ files.
However, here’s the problem (and why things changed recently): some files can’t be replaced until reboot (drivers, etc.). These files are marked as deleted, and will also not show as active.
This means the backup will not collect these files, nor will they ever….the blocks are marked as skipped.
When you perform a full restore, these files that were never backed up will cause a full system restore to fail.
After conferring with MSFT, we stopped skipping the deleted files. Backups are bigger, but will always work during restores.
Let me know if you have any questions as that is a lot of information!
“After conferring with MSFT, we stopped skipping the deleted files. Backups are bigger, but will always work during restores” specific to Commvault or any product using VADP.
Also, when you mention recent; how recent; the backup size grew in September; although the August backups have worked fine. I can check if we installed any hotfix on Commvault recently.
Also, we had Synthetic full backups configured every week; however; I triggered an active full and that actually reduced the backup size by 124GB.
we installed the update on 9th September and we notice this big change on 13th of September;
The update that
@Mike Struening was referring to was included in 11.22.33 under hotfix 3223:
Maintenance Release 11.22.36 (Aug 03, 2021) (commvault.com)
From the information you provided to us, we can see that you upgraded to 11.22.27 to 11.22.36, which would have contained this fix. This would explain the increase in size in the following backups as we would have gone and picked up those deleted blocks which were previously skipped.
Just a question which customer asked about this
Only question I’d have would be if files are deleted, do they continue to get backed up or does it clear out after some time?
Eg say I had 10gb of Outlook PST files that I deleted to free up space on a hard drive, would this continue to get backed up in each backup?:
I could have a 100gb disk and for arguments sake lets say its 100% full. Backup on Monday night takes full 100gb.
Tuesday I then delete 10gb of PST files and replace them with different PST files.
Is Tuesday night backup then 110GB? At which point would the nightly backup be 100GB?
Although this might get into full vs incremental/differential I am guessing.
So being a 30 day retention, if on day 1 backup is 100GB of data, then I delete 10GB and replace with 10GB of different files, I’d expect for 30 days it to have 110GB of backed up data.
After 30 days would this 10GB that was deleted get cleared out of the backup cycle?
I’ll defer to
@Peter Kalinic to confirm what I am writing, though here is my take on this:
Your disk is 100% full (backups are 100GB)
You delete 10 GB of space (backups still 100 GB)
You add more files, say 5 GB) (backups are still 100 GB because we are not discarding the ‘dirty blocks’
I would actually expect that, barring no changes to the vm config, our backup size will always be as big as the vm’s max usage.
I’ll also check with some folks here.