Solved

I have slow Exchange backup/archiving performance – how can I improve it

  • 30 December 2020
  • 4 replies
  • 3309 views

Userlevel 1
Badge +2

I have seen slow performance degradation in my Exchange backups. What can I do to improve speeds?

icon

Best answer by jgeorges 31 December 2020, 05:28

View original

4 replies

Userlevel 5
Badge +9

Hey @japplin ,

Buckle in for a 11 part mini series to share some of the steps we run through here in support when trying to troubleshoot a performance issue:
 

There are many factors which can impact the speed of mailbox backup / archiving job.
Consider this a detailed guide of the troubleshooting steps Commvault Support work through, when investigating performance. 

If these steps don’t manage to identify the cause for performance, they’ll atleast help rule out the most common issues

 

When performing on premise mailbox archiving MAPI is used to access the mailbox data via our mailbox agents. The expected throughput for mailbox backup / archiving is between 2GB/hr - 4GB/hr per stream. It’s worth noting that Exchange Mailbox (Classic) is limited to one stream per job.
 

The V11 Exchange Mailbox agent uses EWS when protecting data from Office 365 instead of MAPI. With EWS, the archiving throughput is increased. 
Note: On Premise archiving still uses MAPI.

 

1) Prerequisites and configuration 

It is important all prerequisites and configuration steps have been followed and implemented for the appropriate backup/archive agent version being used as per the steps in our documentation. Failure to do so can result in low throughput and even failures. Configuration of the throttling policy is essential as without this Exchange throttles connections which can impact throughput. 

Prerequisites Documentation 
V11:  http://documentation.commvault.com/commvault/v11/article?p=products/exchange_mailbox/prerequisites.htm 
V10:  http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/prerequisites.htm 
 

Configuration Documentation 
V11:  http://documentation.commvault.com/commvault/v11/article?p=products/exchange_mailbox/config_basic.htm 
V10:  http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/config_basic.htm 

 

2) Deployment Recommendations 

  • Deployment Planning and Hardware Requirements should be adhered to which can be found in the agent’s documentation. 
  • If possible, an Off-host proxy (client) should be used. This means the agent is installed on a separate computer that is connected to the Exchange servers, in the same site. This method is also known as a Non-Exchange Server installation. This deployment method relieves the load on the memory and processing resources of the Exchange server. However, Commvault Support have seen instances where slow throughput has been improved when the agent was installed on a CAS or Mailbox server, reversing this recommendation. 
  • If the Off-host proxy is a virtual machine, ensure that the VM is reserved enough resources from the host. 
  • The service account's mailbox should be hosted on a mailbox database in the same datacenter as the mailboxes which are being backed up. 

Deployment Documentation 
V11: http://documentation.commvault.com/commvault/v11/article?p=deployment/install/pre_install/r_checklist_ex_mb.htm 
V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/deployment.htm 

 

3) Anti-virus Exclusions 

Ensure all Anti-Virus exclusions are in place as per our documentation. On-access scanning can severely limit the performance of our processes during backup / archive jobs. 
 

Anti-Virus Documentation 
V11: http://documentation.commvault.com/commvault/v11/article?p=white_paper/security/antivirus_exclusions.htm 
V10:  https://documentation.commvault.com/commvault/v10/article?p=white_paper/security/antivirus_exclusions.htm 

 

4) Streams, Subclients and Backup Sets 
For the Exchange Mailbox agent (V11): Multistreaming allows you to archive multiple mailboxes in parallel. By default 5 streams are used. 
 

User Mailbox and Journal Mailbox: Each proxy client that is associated with the Exchange Mailbox client can have a maximum of 20 streams for each job however one service account can only use a maximum of 15 streams regardless of the value set for the subclient. If you need to use a higher number of streams, use more than one service account. 
 

The streams are configured on the subclient properties within the Content of the User Mailbox backupset: 
 

 

 

ContentStore Mailbox: We recommend that you use five streams on each proxy client. 

The streams are configured on the subclient properties within the Content of the ContentStore Mailbox backupset: 

 

 

For the Exchange Mailbox (Classic) agent: Only one mailbox is processed at a time which in turn means only one stream is used throughout the job. If there are many mailboxes to backup/archive and the jobs are not completing in time, multiple subclients can be created within the backupset therefore giving the ability to run multiple jobs in parallel. We recommend to use 5 subclients per mailbox server in the environment. For example, if you have two mailbox role servers, 10 subclients should be created and configured. 
 

Auto discovery subclients can be used to assist in distributing mailboxes across multiple subclients. 

If possible, it is recommended to use Auto Discovery with the Exchange Database Affinity. 

Example 1: 4 mailbox servers, 20 databases & 10,000 mailboxes with 1 proxy client. 

Create 5 sub-clients per each mailbox server and use Exchange Database Affinity Auto-Discovery. 

As there are only 5 sub-clients (5 connections) pointing to each mailbox server, mailbox archive jobs can run throughout the day without any performance impact on exchange server. 

Example 2: 8 mailbox servers, 40 databases & 20,000 mailboxes with 2 proxy clients. 

Create 20 sub-clients per proxy server and use Exchange Database Affinity. 

 

5) Load Balancer Bypass 

Determine if there is a load balancer or DNS round robin in use for Exchange Server CAS / Outlook Anywhere access. If any method of load balancing is in use it may be necessary to bypass this and connect to the Exchange Server directly by using an additional setting or HOSTS file entry. Forcing the use of a specific CAS server may be necessary if one is suspected of having performance issues. 
 

Additional settings: 

Exchange 2010 and below 

Name: sCASServerName 
Category: iDataAgent 
Type: String 
Value: *Exchange CAS Server Name* 
 

Exchange 2013 and above 

Name: sHTTPProxyServer 
Category: iDataAgent 
Type: String 
Value: *Exchange CAS Server Name* 
 

Important Note: If a profile login fails, the CAS server name defined under the sCASServerName or sHTTPProxyServer is deleted. To prevent this from happening, the nDontDeleteExchangeAutoDiscoverKeys additional setting should also be implemented. 

 

6) Disable Exchange Restrictions 

Performance can be improved by changing the search method used to find items to backup/archive. By disabling the use of Exchange Restrictions, the searching for items is performed on the client rather than the Exchange Server. 
 

Exchange Mailbox 
Disable 'Use Exchange restrictions' option in the properties of the Archive and Cleanup Exchange Policy in the CommCell Console: 

Exchange Mailbox (Classic) 
Add the Additional Setting to the client in the CommCell Console as follows: 

Name: nSkipRestriction 
Type: Integer 
Section: MSExchangeMBAgent 
Value: 1 
 

IMPORTANT (For V11 Only): Only enable this additional setting after confirming the following: 
If the backup client (proxy) is SP7 - Hotfix 2081 installed. 
If the backup client (proxy) is SP8 - Hotfix 1973 installed. 
Service packs above V11 SP8 are not effected. 

If the hotfixes are not installed or you want to ensure they are, download the latest hotfix pack for your service pack and deploy this in your CommCell and clients. Hotfix Packs are available at cloud.commvault.com

 

7) Skip backup / tracking of stubs 

Exchange Mailbox 

Movement of stubs between folders within mailboxes can be tracked however this adds some overhead to the archiving jobs. By default the option to do this is disabled however should be checked to see if this is enabled and if so disabled to see if this is impacting performance. 

To do this go to the advanced properties of the Exchange Archiving Policy and disable the "Track stub movement" option in the Advanced tab. 

 Exchange Mailbox (Classic) with OnePass 

Skipping backup of stubs can increase the throughput of archiving. 

Stubs created in one job are backed up in the following job. If a user moves a stub from one folder to another, it will get backed up again and this enables you to restore a message to the new location. If stubs are not backed up, they would be restored to their original location where the item was first archived and stubbed. If this is not an important feature using this additional setting may increase the speed of the job. 
 

To skip backing up stubs, add the nSkipStubs Additional Setting to the client in the CommCell Console as follows: 

Name: nSkipStubs 
Type: Integer 
Section: MSExchangeMBAgent 
Value: 1 

8) Compression and Deduplication 

The backup / archive job performance may be degraded due to compression or deduplication issues. Try running the job without compression and without deduplication to see if performance is improved. 

Disable Compression: 
In the properties of the subclient, navigate to the 'Storage Device' tab and in the 'Data Transfer Option' tab turn Compression off: 

If turning off compression consistently makes a significant impact on the job throughput, resources on the client or media agent should be reviewed depending on where compression is set to occur. 
 

Data Compression Documentation 
V11:  http://documentation.commvault.com/commvault/v11/article?p=features/data_compression/c_data_compression.htm 
V10:  http://documentation.commvault.com/commvault/v10/article?p=features/data_compression/data_compression.htm 
 

Disable Deduplication: 

In the properties of the subclient, navigate to the 'Storage Device' tab and in the 'Deduplication' tab untick 'Enable Deduplication'. 

If disabling deduplication consistently makes a significant impact on the job throughput, the documentation on deduplication should be reviewed to ensure your environment is following best practices and that the correct building blocks are in place. 

Deduplication Overview Documentation 
V11:  http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/c_deduplication_overview.htm 
V10:  http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/c_deduplication_overview.htm 

Deduplication Building Blocks 
V11:  http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/deduplication_building_block.htm 
V10:  http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/deduplication_building_block.htm 

Deduplication Best Practices 
V11:  http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/best_practice.htm 
V10:http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/best_practice.htm 

 

9) TCP Tuning 

TCP Settings in Windows can be tuned to improve throughput. 

  • We would recommend disabling TCP Chimney Offload, RSS and NetDMA on the off-host proxy - see http://support.microsoft.com/kb/951037/en-gb for instructions on how to do this. 
  • Improvements have been achieved by setting TcpAckFrequency to 1 on the off-host proxy. This means TCP packets will be immediately acknowledged rather than the default behaviour of delayed. See the Microsoft KB article http://support.microsoft.com/kb/2020559 and ignore the Hyper-V/iSCSI scenario - All that needs to be achieved is setting it to 1 on the network interface of the off-host proxy. 

 

10) Outlook Network Latency 

High latency between client and Exchange can have a severe impact on the throughput of the backup / archive jobs. The latency can be worked out by subtracting the average processing time from the average response time. We recommend that the network latency value be 20ms or less. 

 

Network Latency = Avg Resp - Avg Proc 
 

To obtain these values log on as the backup service account to the client which performs the backup / archive job: 

1) Open Outlook 
2) Press and hold CTRL and then right-click the Microsoft Outlook icon in the notifications area. 

3) Click Connection Status. 

Avg Proc = Time that the Exchange server spent processing a request 
Avg Resp = Time it took for the request that left Outlook to be responded to and returned back 

Note: Network Latency should never be more than 200ms 

 

Read a Microsoft TechNet blog on Outlook Performance for more information: 
https://blogs.technet.microsoft.com/neiljohn/2012/01/23/outlook-performance-troubleshooting-including-office-365/ 

 

11) Exchange Servers performance 

The performance of the Exchange servers can severely impact mailbox backup / archive jobs. The Exchange servers’ performance should be checked to ensure they are performing as per Microsoft’s requirements. The customer’s Exchange team should check the performance counters to ensure they are not going above the thresholds. If required, the customer can setup Data Collector Sets to monitor the performance counters over time (such as the backup window) and this set can then be reviewed and provided to support if necessary. For more information on Data Collector sets, see https://technet.microsoft.com/en-us/library/cc749337(v=ws.11).aspx 

 

Key performance counters to check which may impact mailbox backup / archive jobs are as follows: 

Exchange Mailbox Servers 

"MSExchangeIS\RPC Requests" - Should always be lower than 70 
"Processor(_Total)\% Processor Time" - Should be less than 75% on average. 
"Memory\% Committed Bytes In Use" - If this value is more than 80% (Exchange 2010) or 90% (Exchange 2013+) it is an indication that the system is likely under stress to provide more memory. 
"MSExchange Database\I/O Database Reads (Attached) Average Latency" - Should be below 20ms 
"MSExchange Database\I/O Database Writes (Attached) Average Latency" - Should be less than Database Reads (Attached) Average Latency when battery-backed write caching is utilized. 

 

References: 
Exchange 2010 - https://technet.microsoft.com/en-us/library/ff367871(v=exchg.141).aspx & https://technet.microsoft.com/en-us/library/ff367896(v=exchg.141).aspx 
Exchange 2013 - https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx 
Exchange 2016 - Microsoft do not provide any documentation on performance counters at this time. 

 

Exchange CAS Servers 

"Processor(_Total)\% Processor Time" - Should be less than 75% on average. 
"Memory\% Committed Bytes In Use" - If this value is more than 80% (Exchange 2010) or 90% (Exchange 2013+) it is an indication that the system is likely under stress to provide more memory. 
"MSExchange RpcClientAccess\RPC Averaged Latency" - Should be less than 250ms 
"MSExchange RpcClientAccess\RPC Requests" - Should be less than 40 

 

References: 
Exchange 2010 - https://technet.microsoft.com/en-us/library/ff367877(v=exchg.141).aspx & https://technet.microsoft.com/en-us/library/ff367896(v=exchg.141).aspx 
Exchange 2013 - https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx 
Exchange 2016 - Microsoft do not provide any documentation on performance counters at this time.

 

-Cheers

Userlevel 4
Badge +11

So i ran into this back in 2020, and mind you we were originally just backing up DB from a DAG. After about 2-3 months the backups for the mailboxes were awful. Taking days for 250 mailboxes, i found that Track stub movement option and removed it- 

All my other check boxes are pretty similar other items highlighted above - but the removal the track stubs made the biggest impact. jobs for those mailboxes increased 600%

Userlevel 5
Badge +10

@jgeorges Great info!!

 

Userlevel 4
Badge +5

Outstanding post @jgeorges!

A few things to add, based on my more theoretical knowledge (vs real-world use and troubleshooting) and working with Dev over the years on questions from clients.

  1. Disable\reduce impact of Throttling: Disabling the Throttling Policy for Exchange - On-Premises (commvault.com)
  2. We support EWS with Exchange 2019, details here on how to enable: Configuring a Service Account to Manage Exchange 2019 Messages Using Exchange Web Services (commvault.com)
    • EWS support for Exchange 2013 and higher I think is in FR23 now too
    • With EXO, I’ve seen throughput rates over 200 GB/hr!
  3. We’ve seen MAPI still out preform EWS with Exchange On-prem
    • I’ve seen rates up to 30GB/hr with a single thread for a single mailbox but there MANY factors that can effect this rate so I normally say 5GB/hr per thread.

Reply