commvault.com commvault.com
Solved

AWS EBS Direct Read API denied

Anthony.Hodges
Rank
Userlevel 5
Badge +10

Greetings,

I have configured AWS backups under FR11.22 and the backups are running as HOTADD instead of using the new EBS Direct Read APIs.

We are using an IAM Role method on the VSA, and the following Allow EBS permissions are have been added for “Resource : *”

"ebs:GetSnapshotBlock",
"ebs:ListChangedBlocks",
"ebs:ListSnapshotBlocks",

However, the follow appears in vsbkp.log.

31965 7d01 04/13 18:23:52 215259 AmazonCompute::CheckPermissions() - Action: [ebs:ListSnapshotBlocks] EvalDecision: [implicitDeny]
31965 7d01 04/13 18:23:52 215259 AmazonCompute::CheckPermissions() - Action: [ebs:GetSnapshotBlock] EvalDecision: [implicitDeny]
31965 7d01 04/13 18:23:52 215259 CAmazonInfo::CheckDirectReadPermissions() - Insufficient permissions for EBS direct read
31965 7d01 04/13 18:23:52 215259 CAmazonInfo::SelectMountMode() - Selected Mode: [HOTADD]
 

What am I missing ? :)

icon

Best answer by Anthony.Hodges 14 April 2021, 10:57

View original

11 replies

S
Rank
Userlevel 7
Badge +15

Hi @Anthony.Hodges 

Here are some things to check, courtesy of the Support team :wink: :

Is EBS direct API supported in the AWS region?

https://docs.aws.amazon.com/general/latest/gr/ebs-service.html

 

Is ec2:DescribeInstanceAttribute enabled?

There are 4 permissions needed, you have mentioned 3, please also check ec2:DescribeInstanceAttribute.

Amazon Web Services Permission Usage

 

Please check the permissions have been granted specifically to this AWS region.

 

The access node needs access to the EBS endpoints.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-accessing-snapshot.html

 

Changed Block Tracking for Amazon

 

Thanks,

Stuart

Customer Support
Anthony.Hodges
Rank
Userlevel 5
Badge +10

Hi @Anthony.Hodges 

Here are some things to check, courtesy of the Support team :wink: :

Is EBS direct API supported in the AWS region?

https://docs.aws.amazon.com/general/latest/gr/ebs-service.html

 

Is ec2:DescribeInstanceAttribute enabled?

There are 4 permissions needed, you have mentioned 3, please also check ec2:DescribeInstanceAttribute.

Amazon Web Services Permission Usage

 

Please check the permissions have been granted specifically to this AWS region.

 

The access node needs access to the EBS endpoints.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-accessing-snapshot.html

 

Changed Block Tracking for Amazon

 

Thanks,

Stuart

 

Is EBS direct API supported in the AWS region?

Yes, ap-southeast-2.

Is ec2:DescribeInstanceAttribute enabled?

Yes.

Please check the permissions have been granted specifically to this AWS region.

In my testing we used Resource *, but can try narrowing to this region.

The access node needs access to the EBS endpoints.

The EBS vpc interface endpoint has not been set up yet whilst doing small VM testing, but it "should" have access.

B
Rank
Userlevel 2
Badge +3

https://documentation.commvault.com/11.22/expert/130078_direct_read_backups_for_amazon.html

Direct read backups use the EBS service, and requires the following AWS permissions:

  • ebs:GetSnapshotBlock

  • iam:SimulatePrincipalPolicy -- Is this permission added too?

Anthony.Hodges
Rank
Userlevel 5
Badge +10

Thanks and yes "iam:SimulatePrincipalPolicy” is set too.

Damian Andre
Rank
Userlevel 7
Badge +23

Couple of other things to check @Anthony.Hodges 

 

  1. Make sure any previously configured registry/additional settings keys for this feature are removed - “bAmazonDirectReadEnabled” under “VirtualServer”
     
  2. Try to disable the check itself with additional setting (it seems that in some cases, API requests could be throttled and therefore giving it a false return that permissions are missing)

    Name: bAmazonSkipDirectReadPermissionsCheck
    Path: VirtualServer
    Type: Boolean
    Value: true
Anthony.Hodges
Rank
Userlevel 5
Badge +10

Ah, sorry I accidentally hit the answer button and I haven't tested it yet. 🤪 Perhaps, a confirmation dialog would be useful for us intermittently cross-eyed/uncoordinated people.

 

Thanks @Damian Andre I will try these.

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@Anthony.Hodges , unmarked for you!

https://www.linkedin.com/in/michael-struening
Anthony.Hodges
Rank
Userlevel 5
Badge +10

I have a support ticket open and it was stated that CV calls simulate-prinicipal-policy for some operations. I suspect that there are additional permissions required to call this API (Testing IAM policies with the IAM policy simulator - AWS Identity and Access Management (amazon.com), & IAM: Access the policy simulator API - AWS Identity and Access Management (amazon.com)).

 

For example, the simulate-principal policy gives an implicit deny to any of the permissions granted, for example S3 list all my buckets.

# aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::11111111111:role/VSA-Role-Name --action-names s3:ListAllMyBuckets

{

    "EvaluationResults": [

        {

            "EvalActionName": "s3:ListAllMyBuckets",

            "EvalResourceName": "*",

            "EvalDecision": "implicitDeny",

            "MatchedStatements": [],

            "MissingContextValues": [],

            "OrganizationsDecisionDetail": {

                "AllowedByOrganizations": false

            }

        }

    ]

}
 

CV Support provided that we can do commands like list snapshot blocks, despite the implicit deny.


# aws ebs list-snapshot-blocks --snapshot-id snap-041e89c92ab0745a3 

{

    "Blocks": [

        {

            "BlockIndex": 0,

            "BlockToken": "AA8BAQ96rUz...

Anthony.Hodges
Rank
Userlevel 5
Badge +10

Sorry, for bothering the Community. The deny reason is at the AWSOrganization SCP level.  We have to get in touch with the account master to resolve this.

S
Rank
Userlevel 7
Badge +15

Hi @Anthony.Hodges 

No worries at all, these discussions are all valuable, showing troubleshooting steps and ideas for the Community to check and follow if anyone else sees something similar in future.

I’ll go ahead and mark your post as Best answer.

Thanks,

Stuart

Customer Support
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Absolutely!  This is no bother, it’s a solid conversation and a great source of information thanks to you, @Anthony.Hodges !

https://www.linkedin.com/in/michael-struening

Reply


Terms & ConditionsCookie settings