Exchange Online performance

  • 2 December 2021
  • 7 replies
  • 722 views

Badge +1

Hello CV community,

I’ve been struggling with throttling issues when archiving EXO and was wondering about your experience in terms of performance.

Could you please share your configuration and the stats of archiving jobs, so I have an idea about what to expect?

  • How many access nodes?
  • How many APP ID’s?
  • How many service accounts?
  • What’s the average throughput when performing archiving jobs?

I know that performance can affected by several factors but I’m hoping go get some general stats with this.

Appreciate your feedback.

Cheers,
Dardan


7 replies

Userlevel 7
Badge +23

Hi @Dardan .  Great question, and welcome to the community :nerd:

As you know, throttling on the Microsoft/Azure side is a constant battle for any o365 backup, though there are definitely ways to improve things!

We do have some best practices docs regarding streams and access nodes here:

https://documentation.commvault.com/11.23/essential/125313_best_practices_for_exchange_online.html

App IDs is dependent on Mailbox count, with a minimum suggested start of 5.

Expected throughput is somewhat impossible to say….too many other factors, though looking at and adjusting the above will certainly improve things.

what are you getting now?  And how many Mailboxes are you protecting?

Badge +1

Hi @Mike Struening ,

Thanks for replying to this threat. I’ve been testing with several setups, have a case open at CV for almost 1 year.

Previous setup:

3 Access Nodes & 1 Index Server (Azure VM’s)
10 App ID’s & 5 Service Accounts (modern auth active)
EXO archives passing through a MA (also in Azure) and going to a AWS bucket

Performance: reaching between up to 100 GB/h BUT encountering throttling (error logs: “Too many concurrent connections opened., Cannot open mailbox”), then streams/pipelines dropping to 0 resulting in no backups.

New setup (just started testing now):

[on-prem] 1 Access node (to start with) with Index Feature installed on it
10 App ID’s & 5 Service Accounts (modern auth active)
EXO archives passing through a MA on-prem going to a local (on-prem) storage library.
So far so good, no throttling encountered BUT performance looks bad to me, around 20GB/h

P.S. in both cases, I ran throttling diagnostics on Microsoft 365 admin center and there is no throttling detected at MS.

Would be interesting to see other people’s configuration and the performance.

Cheers,

Dardan

Userlevel 7
Badge +23

You have a support case open?  Let me know the number and I’ll get some traction!

How many Mailboxes are you protecting?  I have noticed that MSFT’s throttling is obvious in effect more than in declaration…..you see it before anyone says it, but that’s just my experience.

What’s interesting is that you only limited the Access Nodes to 1, everything else is the same, though at the heart, your issue is not just throughput but consistent throughput which is likely the Apps all getting chewed through and MSFT throttling you.

You might max at 20 GB/hour with 1 node.  Would be interesting to add another 10 Apps and see if it increases (I suspect not so much?), then ramp up the Access Nodes and see if it starts to rocket up and STAY there.

Badge +1

Case number: 210412-326

I guess I spoke too soon, 5 hours after starting this job in the new setup, the throttling errors appeared in the logs:

1620  6     12/02 18:56:26 21994974 ### CVFetchMessages::Messages - API: FindItems. Exception: [Too many concurrent connections opened., Cannot open mailbox. server = DB8PR04MB6780.eurprd04.prod.outlook.com, user = /o=ExchangeLabs/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=<recipient removed>, mailbox Guid = <GUI removed>.]. Attempt number: [1]

Correct, our issue is consistent throughput.
If I understand correctly, when using Modern Auth, service accounts are only used to perform discovery then APP ID’s perform the archiving job.

>> Do App ID’s also use EWS to export / archive mailboxes?

We have in total 5977 (including shared, groups/teams mailboxes) in our environment.

Userlevel 7
Badge +23

You are correct on the service account note,  App IDs are everything past that point.

I have a quick review of your case (focused on the past few weeks) and see that you are working with Scott who is an absolutely brilliant legend.  I also saw that our dev team is engaged which is good, though the case is as old as you said.

I’ll reach out to Scott and see what we can do to push this along.

Badge +1

Hi @Dardan,

 

did you found a solution together with Commvault support and dev-team regarding the throughput?

I’m interested as well as we are seeing also poor numbers.

Thanks!

 

Harm

Badge +1

Hi @Harm,

As explained in my previous post, I tested 2 different environment and the configuration of the Previous Setup turned out to have troubles with the proxy being used on the access nodes. CommVault was reporting throttling errors even though services were not throttled by Microsoft, not sure if the proxy stack was being wrong-interpreted by CV.

However with the New Setup there is no (or little) throttling using another type of proxy servers on the access node, and I have an acceptable throughput.

To give you an idea what to expect, with the following environment I am getting an average of 25GB/h for daily incremental backups:

2 access nodes in use
20 App ID’s
~ 6300 mailboxes

For the very first (Full) backup, CommVault support advised to launch several (3 or 4) archiving jobs and we could go up to 50GB/h for each job, so roughly 200GH/h which is not bad.

As it’s been a long time struggle for me, I decided to make a step-by-step doc which explains the installation and configuration of Exchange Online: Protecting O365 Exchange Online with CommVault

Reply