As topic says, are backup and restore of Hyper-V shielded vm’s (Guarded Fabric and Shielded VMs overview | Microsoft Docs) supported ?
And if so, are there any special configuration needed on the Commvault side ?
Solved
Are backup/restore of Hyper-V Shielded VM's supported ?
Best answer by Mike Struening RETIRED
Sharing case solution:
Finding Details:
Goal:
Enable Shielded VMs support for Hyper-V failover cluster.
TPM mode used as the most secure mode - this mode requires special configuration of Hyper-V hosts so health of hosts can be evaluated. As a part of configuration, code integrity policies (in audit mode) with virtualization-based security were enabled.
Issue :
Restore operation of specific VM from each recovery point performed, in mode when we replace existing VM (so backup agent overwrite existing files and re-create VM), both operations failed with error Access Denied. We identified an event was related to VM virtual disk files (VHDX \ VHDX.RCT). For some time, we also couldn’t remove files manually or change any attributes\ACL\owner of file with same error access denied (even under system account). After some time, files disappeared. It looks like operation, or some handles stuck and after some time released.
No specific update on Hyper-v Shielded VM support on COMM vault document
https://www.commvault.com/supported-technologies/microsoft/hyper-v
Commvault backs up the VM data block by block, and not by file by file like a file system backup. So even if the VM is also encrypted, the backup is based on the 'blocks' seen.
This includes the blocks that have changed, and the use of CBT. Please review the following documentation on CBT Here: https://documentation.commvault.com/commvault/v11/article?p=31484.htm
RESTORES:-
1) you will NOT be able to restore granular data from the backed up VM
2) If you do a full VM restore, if the destination host does not have the same Host Guardian Service as the source host, you will not be able to power on the replicated or restored VM.
Reference : https://community.commvault.com/commvault-q-a-2/are-backup-restore-of-hyper-v-shielded-vm-s-supported-2692
Solution:
As per shared logs I do see access error for accessing "Path to VHDX]"
vsrst.log
********************************************************************************************
9788 34b8 03/14 13:35:39 1281534 vsvRstObj::writeMetadata() - CArchiveVirtualDiskFile::Create() failed. Path=[Path to VHDX], error=0x80070005:{CArchiveVirtualDiskFile::Create(1730)} + {CArchiveVirtualDiskFile::CreateVhdFile(1947)} + {CVHDXFileReader::CreateVirtualDisk(1282)/W32.5.(Access is denied. (ERROR_ACCESS_DENIED.5))-CreateVirtualDisk failed for C:\ClusterStorage\vDisk1\oahvcls01\virtual disks\OAHVCLS01.VHDX}
9788 34b8 03/14 13:35:39 1281534 VSRstArchiveReader::ReadPipelineThread() - Error processing buffer type [80]
9788 39e8 03/14 13:35:39 1281534 IDXBROWSECL Callback failed while processing browse error
9788 2e6c 03/14 13:35:39 1281534 SdtTail::onDisconnect: Sending error [98][Services on the tail side of the SDT pipe are going down.] to the head. RCId [19]
9788 2e6c 03/14 13:35:39 1281534 SdtTail::onDisconnect: Sent error code to the head RCId [19]
Tested an “in place” restore, and also 2 “different location” restores, one to a different cluster, where the unshielded vm was fine, the shielded one was not fine (as expected), and also to a different host, in the same cluster, where both restores were fine.
If you have a question or comment, please create a topic
- Vaulter
- 630 replies
Good morning. In the research that I found on this documentation here: https://techcommunity.microsoft.com/t5/data-center-security/what-are-shielded-vms-in-windows-server-2016-hyper-v/ba-p/372179
It doesn't sound like the 'VM' is any different in the Commvault perspective, but that it's a level of encryption for the process to 'turn on the VM' using the Host Guardian Service (HGS), "Without HGS, a Hyper-V host cannot power a Shielded VM on because it can’t decrypt it.".
Commvault backs up the VM data block by block, and not by file by file like a file system backup. So even if the VM is also encrypted, the backup is based on the 'blocks' seen.
This includes the blocks that have changed, and the use of CBT. Please review the following documentation on CBT Here: https://documentation.commvault.com/commvault/v11/article?p=31484.htm
I cannot find any explicate information in the Commvault documentation regarding whether or not CV can backup a Shielded HV VM, however, I have been able to find cases in our archives where Shielded Hyper-V VMs were successfully protected.
I think we can expect the backups will work, but based on the HGS service, it possibly sounds like Application-Aware and/or File System and Application Consistent backups may not be available.
What I would also consider here, is how the RESTORES will work. Because you're working with these shielded/encrypted VMs, you may find:
1) that you will NOT be able to restore granular data from the backed up VM
2) If you do a full VM restore, if the destination host does not have the same Host Guardian Service as the source host, you will not be able to power on the replicated or restored VM.
As always, I would recommend testing this in a lab or Dev environment prior to rolling out to your production environment.
Please let me know if you have any further questions
We did do a little bit of quick testing, which is what brought me to make this post. Because as you said, it seems that backups worked “as normal”, but we had issued with restores. It seemed to complain about “access denied” to the file/folders where the original VM were.
I _think_ (but not 100% sure) that we were able to do a restore, but only when it was restored to a different location (on the same hyper-v host)… which is a bit painful.
And the Microsoft engineer involved from Microsoft’s side had some “questions”, and possible ideas as to why this might have happened,
So i’ve created a support-case with those, and hopefully we’ll figure out how this should work… and configure anything needed, or change operational routines to also support different ways of doing backup/restore.
+8- Vaulter
- 35 replies
Just a thought to throw out there as I don't think it has been mentioned on the thread yet - For granular restore capabilities you could use an in-guest agent and run a traditional backup in addition to the VSA image based backup.
1 person likes this
+23- Vaulter
- 4996 replies
- Answer
Sharing case solution:
Finding Details:
Goal:
Enable Shielded VMs support for Hyper-V failover cluster.
TPM mode used as the most secure mode - this mode requires special configuration of Hyper-V hosts so health of hosts can be evaluated. As a part of configuration, code integrity policies (in audit mode) with virtualization-based security were enabled.
Issue :
Restore operation of specific VM from each recovery point performed, in mode when we replace existing VM (so backup agent overwrite existing files and re-create VM), both operations failed with error Access Denied. We identified an event was related to VM virtual disk files (VHDX \ VHDX.RCT). For some time, we also couldn’t remove files manually or change any attributes\ACL\owner of file with same error access denied (even under system account). After some time, files disappeared. It looks like operation, or some handles stuck and after some time released.
No specific update on Hyper-v Shielded VM support on COMM vault document
https://www.commvault.com/supported-technologies/microsoft/hyper-v
Commvault backs up the VM data block by block, and not by file by file like a file system backup. So even if the VM is also encrypted, the backup is based on the 'blocks' seen.
This includes the blocks that have changed, and the use of CBT. Please review the following documentation on CBT Here: https://documentation.commvault.com/commvault/v11/article?p=31484.htm
RESTORES:-
1) you will NOT be able to restore granular data from the backed up VM
2) If you do a full VM restore, if the destination host does not have the same Host Guardian Service as the source host, you will not be able to power on the replicated or restored VM.
Reference : https://community.commvault.com/commvault-q-a-2/are-backup-restore-of-hyper-v-shielded-vm-s-supported-2692
Solution:
As per shared logs I do see access error for accessing "Path to VHDX]"
vsrst.log
********************************************************************************************
9788 34b8 03/14 13:35:39 1281534 vsvRstObj::writeMetadata() - CArchiveVirtualDiskFile::Create() failed. Path=[Path to VHDX], error=0x80070005:{CArchiveVirtualDiskFile::Create(1730)} + {CArchiveVirtualDiskFile::CreateVhdFile(1947)} + {CVHDXFileReader::CreateVirtualDisk(1282)/W32.5.(Access is denied. (ERROR_ACCESS_DENIED.5))-CreateVirtualDisk failed for C:\ClusterStorage\vDisk1\oahvcls01\virtual disks\OAHVCLS01.VHDX}
9788 34b8 03/14 13:35:39 1281534 VSRstArchiveReader::ReadPipelineThread() - Error processing buffer type [80]
9788 39e8 03/14 13:35:39 1281534 IDXBROWSECL Callback failed while processing browse error
9788 2e6c 03/14 13:35:39 1281534 SdtTail::onDisconnect: Sending error [98][Services on the tail side of the SDT pipe are going down.] to the head. RCId [19]
9788 2e6c 03/14 13:35:39 1281534 SdtTail::onDisconnect: Sent error code to the head RCId [19]
Tested an “in place” restore, and also 2 “different location” restores, one to a different cluster, where the unshielded vm was fine, the shielded one was not fine (as expected), and also to a different host, in the same cluster, where both restores were fine.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.