Solved

Using ClusterSharedVolumes with Ransomware Protection ?


Userlevel 1
Badge +4

Hi,
Is there a way to use Ransomeware Protection on Windows MediaAgents, using a Disklibrary with Cluster Share Volumes ?

Once Ransomeware Protection is activated, the filter driver “CVDLP” with the altitude of 145180 (encryption) is added to the Filesystem Filter.

this results in redirected I/Os to all  ClusterSharedVolumes :

BlockRedirectedIOReason      : NotBlockRedirected

FileSystemRedirectedIOReason : IncompatibleFileSystemFilter

Name                         : volume21

Node                         : node1

StateInfo                    : FileSystemRedirected

as a result the Clusted Events are flooded with Warnings:

Cluster Shared Volume 'volume21' ('volume21') has identified one or more active filter drivers on this device stack that could interfere with CSV operations. I/O access will be redirected to the storage device over the network through another Cluster node. This may result in degraded performance. Please contact the filter driver vendor to verify interoperability with Cluster Shared Volumes. 

Active filter drivers found:

CVDLP (Encryption)

even though I don’t experience any delay compared to Media access w/o Ransomeware Protection, I’d like to get rid of the warnings at least.

BR

Klaus

icon

Best answer by johanningk 28 April 2021, 16:07

good news.

I found the solution :

  • disable ransomware protection on all MAs in the Cluster
  • check for CVDLP filter using fltmc.exe
  • wait until all CSVs are reported to have direct access using 

Get-ClusterSharedVolumeState |where FileSystemRedirectedIOReason -like 'IncompatibleFileSystemFilter'|ft name,node,StateInfo

  • re-enable ransomware protection on all MAs in the Cluster

The MR Update didn’t seem to un-/reload the CVDLP filter during the Patch installation.
since these server have not been rebooted since this step ……

now everthings looks fine 

thanks

View original

18 replies

Userlevel 1
Badge +2

Hi Klaus,

     What service pack level do you have installed?  This is fixed in SP20 and above, although the fix is somewhat recent so you may not have it yet.  Otherwise, we have a manual workaround that i can send you.  

Userlevel 1
Badge +4

Hi Brent,

that’s great news, currently the CommCell in question is still on SP16 but I plan to upgrade to SP20 (LTS) in April.

I’m glad that this is solved 

I’ll give it a try on another CommCell, currently on FR21 already :)

Userlevel 1
Badge +2

Hi,

     On SP21 you will need maintenance release 30 (so 11.21.30).  The HF number is 2354.  While the hotfix itself is available, the maintenance release its included in most likely will not be made available until the first week of April.  If you would like to upgrade your SP21 to the latest maintenance release (11.21.27), i can provide you a bundle of HF 2354 that you can install on top of it.  Or I can give you the manual workaround for now.  Let me know how you’d like to proceed.

Userlevel 1
Badge +4

Hi Brent,

can you provide the HF number and MR for FR20 as well ?
I could not find a reference in BOL for these patches on FR20 - 22.
Most of the environments I have to handle are on FR20 and will not be upgraded until FR24 (LTS) is available.
if not required because of new features or product version support, customers are fairly conservative :)

Userlevel 1
Badge +2

sp20 HF: 3482 (this will be in 11.20.42)

by the way, i gave you the wrong HF number for SP21 (i gave you the form id, not the HF number). 

sp21 HF: 2350 (this will be in 11.21.30)

Userlevel 1
Badge +4

Thanks Brent,

I keep you updated, once I was able to implement the Hotfix :slight_smile:

Userlevel 1
Badge +4

Hi,
I can comfirm, that the latest MR (46) on FR20 includes a patch, that successfully suppress the Cluster Shared Volume Redirection events on the Microsoft Cluster.

I also have an environment updated to FR22, but the Hotfix doesn’t seem to be included in the latest MR there.

I hope it will be added to the next one …..

Userlevel 1
Badge +2

on FR22, the HF number is 2228.  Its in MR 16, so 22.16 and above should have it.

Userlevel 1
Badge +4

I’m on V11.22.18 (even though this might have been pulled back and replaced by V11.22.17).

The mentioned HF is part of MR18, but does not show the desired effect.
I can find it in the list of installed HF on the MediaAgent(s).

System Event Logs are still flooded with EventId 5125, while this is no longer the case on the V11.20.46 systems. 

 

Userlevel 1
Badge +2

Hmm.  Is there any chance one or more of the nodes that can access the Cluster Shared Volumes doesnt have the fix?  If all nodes have the fix, you can create a customer ticket and i’ll check it out (would need logs and csdb)

Userlevel 1
Badge +4

Hi Brent,

all nodes connected to the CSVs have the same software version.
I’ll raise a ticket tomorrow, since this is not expected behavior.

Thanks

Userlevel 1
Badge +4

good news.

I found the solution :

  • disable ransomware protection on all MAs in the Cluster
  • check for CVDLP filter using fltmc.exe
  • wait until all CSVs are reported to have direct access using 

Get-ClusterSharedVolumeState |where FileSystemRedirectedIOReason -like 'IncompatibleFileSystemFilter'|ft name,node,StateInfo

  • re-enable ransomware protection on all MAs in the Cluster

The MR Update didn’t seem to un-/reload the CVDLP filter during the Patch installation.
since these server have not been rebooted since this step ……

now everthings looks fine 

thanks

Userlevel 1
Badge +2

Ahhh OK, glad to know that fixed it. 

Badge

Hi Klaus,

     What service pack level do you have installed?  This is fixed in SP20 and above, although the fix is somewhat recent so you may not have it yet.  Otherwise, we have a manual workaround that i can send you.  


Hi Brent,

We have a customer with the exact same issue, and they have requested the manual workaround. I don’t suppose you could reveal what the manual workaround is?

Thanks

Jason

Userlevel 1
Badge +4

Hi Jason, 
as mentioned earlier in this Thread, the required fixes to Commvault CVFLT driver (or MS Cluster reaction, when finding them) have been provided with:
 

V11.20.42
V11.21.30
V11.22.16

And later
I don’t see any more problems, when activating ransomware protection on Cluster Shared Volumes.
latest installation done, is running on V11.23.10 and doesn’t show any issues as well.

have fun
Klaus

Badge

Hi Klaus,

Yes - the issue we have is that the customer doesn’t have plans to their update their service pack level at the moment. There are plans in place to do this, but can’t give a timeframe as to when it will be done. 

The customer has also seen this thread (they pointed it out to us) so are aware there is a manual workaround available for versions below the ones you’ve described, hence why we have requested it from Brent.

Thanks

Jason

Userlevel 1
Badge +2

Hi Jason,

     Can you access this link?  It has all the details on the workaround.

https://alexandria.commvault.com/article/details/55030/1536

Userlevel 7
Badge +17

@JasonBush I edited the article and added Partner access.  Let me know if you have issues accessing:

https://alexandria.commvault.com/article/details/55030

Reply