I have seen slow performance degradation in my Exchange backups. What can I do to improve speeds?
Hey
Buckle in for a 11 part mini series to share some of the steps we run through here in support when trying to troubleshoot a performance issue:
There are many factors which can impact the speed of mailbox backup / archiving job.
Consider this a detailed guide of the troubleshooting steps Commvault Support work through, when investigating performance.
If these steps don’t manage to identify the cause for performance, they’ll atleast help rule out the most common issues
When performing on premise mailbox archiving MAPI is used to access the mailbox data via our mailbox agents. The expected throughput for mailbox backup / archiving is between 2GB/hr - 4GB/hr per stream. It’s worth noting that Exchange Mailbox (Classic) is limited to one stream per job.
The V11 Exchange Mailbox agent uses EWS when protecting data from Office 365 instead of MAPI. With EWS, the archiving throughput is increased.
Note: On Premise archiving still uses MAPI.
1) Prerequisites and configuration
It is important all prerequisites and configuration steps have been followed and implemented for the appropriate backup/archive agent version being used as per the steps in our documentation. Failure to do so can result in low throughput and even failures. Configuration of the throttling policy is essential as without this Exchange throttles connections which can impact throughput.
Prerequisites Documentation
V11: http://documentation.commvault.com/commvault/v11/article?p=products/exchange_mailbox/prerequisites.htm
V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/prerequisites.htm
Configuration Documentation
V11: http://documentation.commvault.com/commvault/v11/article?p=products/exchange_mailbox/config_basic.htm
V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/config_basic.htm
2) Deployment Recommendations
- Deployment Planning and Hardware Requirements should be adhered to which can be found in the agent’s documentation.
- If possible, an Off-host proxy (client) should be used. This means the agent is installed on a separate computer that is connected to the Exchange servers, in the same site. This method is also known as a Non-Exchange Server installation. This deployment method relieves the load on the memory and processing resources of the Exchange server. However, Commvault Support have seen instances where slow throughput has been improved when the agent was installed on a CAS or Mailbox server, reversing this recommendation.
- If the Off-host proxy is a virtual machine, ensure that the VM is reserved enough resources from the host.
- The service account's mailbox should be hosted on a mailbox database in the same datacenter as the mailboxes which are being backed up.
Deployment Documentation
V11: http://documentation.commvault.com/commvault/v11/article?p=deployment/install/pre_install/r_checklist_ex_mb.htm
V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/deployment.htm
3) Anti-virus Exclusions
Ensure all Anti-Virus exclusions are in place as per our documentation. On-access scanning can severely limit the performance of our processes during backup / archive jobs.
Anti-Virus Documentation
V11: http://documentation.commvault.com/commvault/v11/article?p=white_paper/security/antivirus_exclusions.htm
V10: https://documentation.commvault.com/commvault/v10/article?p=white_paper/security/antivirus_exclusions.htm
4) Streams, Subclients and Backup Sets
For the Exchange Mailbox agent (V11): Multistreaming allows you to archive multiple mailboxes in parallel. By default 5 streams are used.
User Mailbox and Journal Mailbox: Each proxy client that is associated with the Exchange Mailbox client can have a maximum of 20 streams for each job however one service account can only use a maximum of 15 streams regardless of the value set for the subclient. If you need to use a higher number of streams, use more than one service account.
The streams are configured on the subclient properties within the Content of the User Mailbox backupset:
ContentStore Mailbox: We recommend that you use five streams on each proxy client.
The streams are configured on the subclient properties within the Content of the ContentStore Mailbox backupset:
For the Exchange Mailbox (Classic) agent: Only one mailbox is processed at a time which in turn means only one stream is used throughout the job. If there are many mailboxes to backup/archive and the jobs are not completing in time, multiple subclients can be created within the backupset therefore giving the ability to run multiple jobs in parallel. We recommend to use 5 subclients per mailbox server in the environment. For example, if you have two mailbox role servers, 10 subclients should be created and configured.
Auto discovery subclients can be used to assist in distributing mailboxes across multiple subclients.
If possible, it is recommended to use Auto Discovery with the Exchange Database Affinity.
Example 1: 4 mailbox servers, 20 databases & 10,000 mailboxes with 1 proxy client.
Create 5 sub-clients per each mailbox server and use Exchange Database Affinity Auto-Discovery.
As there are only 5 sub-clients (5 connections) pointing to each mailbox server, mailbox archive jobs can run throughout the day without any performance impact on exchange server.
Example 2: 8 mailbox servers, 40 databases & 20,000 mailboxes with 2 proxy clients.
Create 20 sub-clients per proxy server and use Exchange Database Affinity.
5) Load Balancer Bypass
Determine if there is a load balancer or DNS round robin in use for Exchange Server CAS / Outlook Anywhere access. If any method of load balancing is in use it may be necessary to bypass this and connect to the Exchange Server directly by using an additional setting or HOSTS file entry. Forcing the use of a specific CAS server may be necessary if one is suspected of having performance issues.
Additional settings:
Exchange 2010 and below
Name: sCASServerName
Category: iDataAgent
Type: String
Value: *Exchange CAS Server Name*
Exchange 2013 and above
Name: sHTTPProxyServer
Category: iDataAgent
Type: String
Value: *Exchange CAS Server Name*
Important Note: If a profile login fails, the CAS server name defined under the sCASServerName or sHTTPProxyServer is deleted. To prevent this from happening, the nDontDeleteExchangeAutoDiscoverKeys additional setting should also be implemented.
6) Disable Exchange Restrictions
Performance can be improved by changing the search method used to find items to backup/archive. By disabling the use of Exchange Restrictions, the searching for items is performed on the client rather than the Exchange Server.
Exchange Mailbox
Disable 'Use Exchange restrictions' option in the properties of the Archive and Cleanup Exchange Policy in the CommCell Console:
Exchange Mailbox (Classic)
Add the Additional Setting to the client in the CommCell Console as follows:
Name: nSkipRestriction
Type: Integer
Section: MSExchangeMBAgent
Value: 1
IMPORTANT (For V11 Only): Only enable this additional setting after confirming the following:
If the backup client (proxy) is SP7 - Hotfix 2081 installed.
If the backup client (proxy) is SP8 - Hotfix 1973 installed.
Service packs above V11 SP8 are not effected.
If the hotfixes are not installed or you want to ensure they are, download the latest hotfix pack for your service pack and deploy this in your CommCell and clients. Hotfix Packs are available at cloud.commvault.com.
7) Skip backup / tracking of stubs
Exchange Mailbox
Movement of stubs between folders within mailboxes can be tracked however this adds some overhead to the archiving jobs. By default the option to do this is disabled however should be checked to see if this is enabled and if so disabled to see if this is impacting performance.
To do this go to the advanced properties of the Exchange Archiving Policy and disable the "Track stub movement" option in the Advanced tab.
Exchange Mailbox (Classic) with OnePass
Skipping backup of stubs can increase the throughput of archiving.
Stubs created in one job are backed up in the following job. If a user moves a stub from one folder to another, it will get backed up again and this enables you to restore a message to the new location. If stubs are not backed up, they would be restored to their original location where the item was first archived and stubbed. If this is not an important feature using this additional setting may increase the speed of the job.
To skip backing up stubs, add the nSkipStubs Additional Setting to the client in the CommCell Console as follows:
Name: nSkipStubs
Type: Integer
Section: MSExchangeMBAgent
Value: 1
8) Compression and Deduplication
The backup / archive job performance may be degraded due to compression or deduplication issues. Try running the job without compression and without deduplication to see if performance is improved.
Disable Compression:
In the properties of the subclient, navigate to the 'Storage Device' tab and in the 'Data Transfer Option' tab turn Compression off:
If turning off compression consistently makes a significant impact on the job throughput, resources on the client or media agent should be reviewed depending on where compression is set to occur.
Data Compression Documentation
V11: http://documentation.commvault.com/commvault/v11/article?p=features/data_compression/c_data_compression.htm
V10: http://documentation.commvault.com/commvault/v10/article?p=features/data_compression/data_compression.htm
Disable Deduplication:
In the properties of the subclient, navigate to the 'Storage Device' tab and in the 'Deduplication' tab untick 'Enable Deduplication'.
If disabling deduplication consistently makes a significant impact on the job throughput, the documentation on deduplication should be reviewed to ensure your environment is following best practices and that the correct building blocks are in place.
Deduplication Overview Documentation
V11: http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/c_deduplication_overview.htm
V10: http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/c_deduplication_overview.htm
Deduplication Building Blocks
V11: http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/deduplication_building_block.htm
V10: http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/deduplication_building_block.htm
Deduplication Best Practices
V11: http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/best_practice.htm
V10:http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/best_practice.htm
9) TCP Tuning
TCP Settings in Windows can be tuned to improve throughput.
- We would recommend disabling TCP Chimney Offload, RSS and NetDMA on the off-host proxy - see http://support.microsoft.com/kb/951037/en-gb for instructions on how to do this.
- Improvements have been achieved by setting TcpAckFrequency to 1 on the off-host proxy. This means TCP packets will be immediately acknowledged rather than the default behaviour of delayed. See the Microsoft KB article http://support.microsoft.com/kb/2020559 and ignore the Hyper-V/iSCSI scenario - All that needs to be achieved is setting it to 1 on the network interface of the off-host proxy.
10) Outlook Network Latency
High latency between client and Exchange can have a severe impact on the throughput of the backup / archive jobs. The latency can be worked out by subtracting the average processing time from the average response time. We recommend that the network latency value be 20ms or less.
Network Latency = Avg Resp - Avg Proc
To obtain these values log on as the backup service account to the client which performs the backup / archive job:
1) Open Outlook
2) Press and hold CTRL and then right-click the Microsoft Outlook icon in the notifications area.
3) Click Connection Status.
Avg Proc = Time that the Exchange server spent processing a request
Avg Resp = Time it took for the request that left Outlook to be responded to and returned back
Note: Network Latency should never be more than 200ms
Read a Microsoft TechNet blog on Outlook Performance for more information:
https://blogs.technet.microsoft.com/neiljohn/2012/01/23/outlook-performance-troubleshooting-including-office-365/
11) Exchange Servers performance
The performance of the Exchange servers can severely impact mailbox backup / archive jobs. The Exchange servers’ performance should be checked to ensure they are performing as per Microsoft’s requirements. The customer’s Exchange team should check the performance counters to ensure they are not going above the thresholds. If required, the customer can setup Data Collector Sets to monitor the performance counters over time (such as the backup window) and this set can then be reviewed and provided to support if necessary. For more information on Data Collector sets, see https://technet.microsoft.com/en-us/library/cc749337(v=ws.11).aspx
Key performance counters to check which may impact mailbox backup / archive jobs are as follows:
Exchange Mailbox Servers
"MSExchangeIS\RPC Requests" - Should always be lower than 70
"Processor(_Total)\% Processor Time" - Should be less than 75% on average.
"Memory\% Committed Bytes In Use" - If this value is more than 80% (Exchange 2010) or 90% (Exchange 2013+) it is an indication that the system is likely under stress to provide more memory.
"MSExchange Database\I/O Database Reads (Attached) Average Latency" - Should be below 20ms
"MSExchange Database\I/O Database Writes (Attached) Average Latency" - Should be less than Database Reads (Attached) Average Latency when battery-backed write caching is utilized.
References:
Exchange 2010 - https://technet.microsoft.com/en-us/library/ff367871(v=exchg.141).aspx & https://technet.microsoft.com/en-us/library/ff367896(v=exchg.141).aspx
Exchange 2013 - https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx
Exchange 2016 - Microsoft do not provide any documentation on performance counters at this time.
Exchange CAS Servers
"Processor(_Total)\% Processor Time" - Should be less than 75% on average.
"Memory\% Committed Bytes In Use" - If this value is more than 80% (Exchange 2010) or 90% (Exchange 2013+) it is an indication that the system is likely under stress to provide more memory.
"MSExchange RpcClientAccess\RPC Averaged Latency" - Should be less than 250ms
"MSExchange RpcClientAccess\RPC Requests" - Should be less than 40
References:
Exchange 2010 - https://technet.microsoft.com/en-us/library/ff367877(v=exchg.141).aspx & https://technet.microsoft.com/en-us/library/ff367896(v=exchg.141).aspx
Exchange 2013 - https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx
Exchange 2016 - Microsoft do not provide any documentation on performance counters at this time.
-Cheers
So i ran into this back in 2020, and mind you we were originally just backing up DB from a DAG. After about 2-3 months the backups for the mailboxes were awful. Taking days for 250 mailboxes, i found that Track stub movement option and removed it-
All my other check boxes are pretty similar other items highlighted above - but the removal the track stubs made the biggest impact. jobs for those mailboxes increased 600%
Outstanding post @jgeorges!
A few things to add, based on my more theoretical knowledge (vs real-world use and troubleshooting) and working with Dev over the years on questions from clients.
- Disable\reduce impact of Throttling: Disabling the Throttling Policy for Exchange - On-Premises (commvault.com)
- We support EWS with Exchange 2019, details here on how to enable: Configuring a Service Account to Manage Exchange 2019 Messages Using Exchange Web Services (commvault.com)
- EWS support for Exchange 2013 and higher I think is in FR23 now too
- With EXO, I’ve seen throughput rates over 200 GB/hr!
- We’ve seen MAPI still out preform EWS with Exchange On-prem
- I’ve seen rates up to 30GB/hr with a single thread for a single mailbox but there MANY factors that can effect this rate so I normally say 5GB/hr per thread.
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.