I have seen slow performance degradation in my Exchange backups. What can I do to improve speeds?
Best answer by jgeorges 31 December 2020, 05:28
Hey @japplin ,
Buckle in for a 11 part mini series to share some of the steps we run through here in support when trying to troubleshoot a performance issue:
There are many factors which can impact the speed of mailbox backup / archiving job.Consider this a detailed guide of the troubleshooting steps Commvault Support work through, when investigating performance.
If these steps don’t manage to identify the cause for performance, they’ll atleast help rule out the most common issues
When performing on premise mailbox archiving MAPI is used to access the mailbox data via our mailbox agents. The expected throughput for mailbox backup / archiving is between 2GB/hr - 4GB/hr per stream. It’s worth noting that Exchange Mailbox (Classic) is limited to one stream per job.
The V11 Exchange Mailbox agent uses EWS when protecting data from Office 365 instead of MAPI. With EWS, the archiving throughput is increased. Note: On Premise archiving still uses MAPI.
1) Prerequisites and configuration
It is important all prerequisites and configuration steps have been followed and implemented for the appropriate backup/archive agent version being used as per the steps in our documentation. Failure to do so can result in low throughput and even failures. Configuration of the throttling policy is essential as without this Exchange throttles connections which can impact throughput.
Prerequisites Documentation V11: http://documentation.commvault.com/commvault/v11/article?p=products/exchange_mailbox/prerequisites.htm V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/prerequisites.htm
Configuration Documentation V11: http://documentation.commvault.com/commvault/v11/article?p=products/exchange_mailbox/config_basic.htm V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/config_basic.htm
2) Deployment Recommendations
Deployment Documentation V11: http://documentation.commvault.com/commvault/v11/article?p=deployment/install/pre_install/r_checklist_ex_mb.htm V10: http://documentation.commvault.com/commvault/v10/article?p=products/exchange_mailbox/deployment.htm
3) Anti-virus Exclusions
Ensure all Anti-Virus exclusions are in place as per our documentation. On-access scanning can severely limit the performance of our processes during backup / archive jobs.
Anti-Virus Documentation V11: http://documentation.commvault.com/commvault/v11/article?p=white_paper/security/antivirus_exclusions.htm V10: https://documentation.commvault.com/commvault/v10/article?p=white_paper/security/antivirus_exclusions.htm
4) Streams, Subclients and Backup Sets For the Exchange Mailbox agent (V11): Multistreaming allows you to archive multiple mailboxes in parallel. By default 5 streams are used.
User Mailbox and Journal Mailbox: Each proxy client that is associated with the Exchange Mailbox client can have a maximum of 20 streams for each job however one service account can only use a maximum of 15 streams regardless of the value set for the subclient. If you need to use a higher number of streams, use more than one service account.
The streams are configured on the subclient properties within the Content of the User Mailbox backupset:
ContentStore Mailbox: We recommend that you use five streams on each proxy client.
The streams are configured on the subclient properties within the Content of the ContentStore Mailbox backupset:
For the Exchange Mailbox (Classic) agent: Only one mailbox is processed at a time which in turn means only one stream is used throughout the job. If there are many mailboxes to backup/archive and the jobs are not completing in time, multiple subclients can be created within the backupset therefore giving the ability to run multiple jobs in parallel. We recommend to use 5 subclients per mailbox server in the environment. For example, if you have two mailbox role servers, 10 subclients should be created and configured.
Auto discovery subclients can be used to assist in distributing mailboxes across multiple subclients.
If possible, it is recommended to use Auto Discovery with the Exchange Database Affinity.
Example 1: 4 mailbox servers, 20 databases & 10,000 mailboxes with 1 proxy client.
Create 5 sub-clients per each mailbox server and use Exchange Database Affinity Auto-Discovery.
As there are only 5 sub-clients (5 connections) pointing to each mailbox server, mailbox archive jobs can run throughout the day without any performance impact on exchange server.
Example 2: 8 mailbox servers, 40 databases & 20,000 mailboxes with 2 proxy clients.
Create 20 sub-clients per proxy server and use Exchange Database Affinity.
5) Load Balancer Bypass
Determine if there is a load balancer or DNS round robin in use for Exchange Server CAS / Outlook Anywhere access. If any method of load balancing is in use it may be necessary to bypass this and connect to the Exchange Server directly by using an additional setting or HOSTS file entry. Forcing the use of a specific CAS server may be necessary if one is suspected of having performance issues.
Exchange 2010 and below
Name: sCASServerName Category: iDataAgent Type: String Value: *Exchange CAS Server Name*
Exchange 2013 and above
Name: sHTTPProxyServer Category: iDataAgent Type: String Value: *Exchange CAS Server Name*
Important Note: If a profile login fails, the CAS server name defined under the sCASServerName or sHTTPProxyServer is deleted. To prevent this from happening, the nDontDeleteExchangeAutoDiscoverKeys additional setting should also be implemented.
6) Disable Exchange Restrictions
Performance can be improved by changing the search method used to find items to backup/archive. By disabling the use of Exchange Restrictions, the searching for items is performed on the client rather than the Exchange Server.
Exchange Mailbox Disable 'Use Exchange restrictions' option in the properties of the Archive and Cleanup Exchange Policy in the CommCell Console:
Exchange Mailbox (Classic) Add the Additional Setting to the client in the CommCell Console as follows:
Name: nSkipRestriction Type: Integer Section: MSExchangeMBAgent Value: 1
IMPORTANT (For V11 Only): Only enable this additional setting after confirming the following: If the backup client (proxy) is SP7 - Hotfix 2081 installed. If the backup client (proxy) is SP8 - Hotfix 1973 installed. Service packs above V11 SP8 are not effected.
If the hotfixes are not installed or you want to ensure they are, download the latest hotfix pack for your service pack and deploy this in your CommCell and clients. Hotfix Packs are available at cloud.commvault.com.
7) Skip backup / tracking of stubs
Movement of stubs between folders within mailboxes can be tracked however this adds some overhead to the archiving jobs. By default the option to do this is disabled however should be checked to see if this is enabled and if so disabled to see if this is impacting performance.
To do this go to the advanced properties of the Exchange Archiving Policy and disable the "Track stub movement" option in the Advanced tab.
Exchange Mailbox (Classic) with OnePass
Skipping backup of stubs can increase the throughput of archiving.
Stubs created in one job are backed up in the following job. If a user moves a stub from one folder to another, it will get backed up again and this enables you to restore a message to the new location. If stubs are not backed up, they would be restored to their original location where the item was first archived and stubbed. If this is not an important feature using this additional setting may increase the speed of the job.
To skip backing up stubs, add the nSkipStubs Additional Setting to the client in the CommCell Console as follows:
Name: nSkipStubs Type: Integer Section: MSExchangeMBAgent Value: 1
8) Compression and Deduplication
The backup / archive job performance may be degraded due to compression or deduplication issues. Try running the job without compression and without deduplication to see if performance is improved.
Disable Compression: In the properties of the subclient, navigate to the 'Storage Device' tab and in the 'Data Transfer Option' tab turn Compression off:
If turning off compression consistently makes a significant impact on the job throughput, resources on the client or media agent should be reviewed depending on where compression is set to occur.
Data Compression Documentation V11: http://documentation.commvault.com/commvault/v11/article?p=features/data_compression/c_data_compression.htm V10: http://documentation.commvault.com/commvault/v10/article?p=features/data_compression/data_compression.htm
In the properties of the subclient, navigate to the 'Storage Device' tab and in the 'Deduplication' tab untick 'Enable Deduplication'.
If disabling deduplication consistently makes a significant impact on the job throughput, the documentation on deduplication should be reviewed to ensure your environment is following best practices and that the correct building blocks are in place.
Deduplication Overview Documentation V11: http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/c_deduplication_overview.htm V10: http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/c_deduplication_overview.htm
Deduplication Building Blocks V11: http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/deduplication_building_block.htm V10: http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/deduplication_building_block.htm
Deduplication Best Practices V11: http://documentation.commvault.com/commvault/v11/article?p=features/deduplication/best_practice.htm V10:http://documentation.commvault.com/commvault/v10/article?p=features/deduplication/best_practice.htm
9) TCP Tuning
TCP Settings in Windows can be tuned to improve throughput.
10) Outlook Network Latency
High latency between client and Exchange can have a severe impact on the throughput of the backup / archive jobs. The latency can be worked out by subtracting the average processing time from the average response time. We recommend that the network latency value be 20ms or less.
Network Latency = Avg Resp - Avg Proc
To obtain these values log on as the backup service account to the client which performs the backup / archive job:
1) Open Outlook 2) Press and hold CTRL and then right-click the Microsoft Outlook icon in the notifications area.
3) Click Connection Status.
Avg Proc = Time that the Exchange server spent processing a request Avg Resp = Time it took for the request that left Outlook to be responded to and returned back
Note: Network Latency should never be more than 200ms
Read a Microsoft TechNet blog on Outlook Performance for more information: https://blogs.technet.microsoft.com/neiljohn/2012/01/23/outlook-performance-troubleshooting-including-office-365/
11) Exchange Servers performance
The performance of the Exchange servers can severely impact mailbox backup / archive jobs. The Exchange servers’ performance should be checked to ensure they are performing as per Microsoft’s requirements. The customer’s Exchange team should check the performance counters to ensure they are not going above the thresholds. If required, the customer can setup Data Collector Sets to monitor the performance counters over time (such as the backup window) and this set can then be reviewed and provided to support if necessary. For more information on Data Collector sets, see https://technet.microsoft.com/en-us/library/cc749337(v=ws.11).aspx
Key performance counters to check which may impact mailbox backup / archive jobs are as follows:
Exchange Mailbox Servers
"MSExchangeIS\RPC Requests" - Should always be lower than 70 "Processor(_Total)\% Processor Time" - Should be less than 75% on average. "Memory\% Committed Bytes In Use" - If this value is more than 80% (Exchange 2010) or 90% (Exchange 2013+) it is an indication that the system is likely under stress to provide more memory. "MSExchange Database\I/O Database Reads (Attached) Average Latency" - Should be below 20ms "MSExchange Database\I/O Database Writes (Attached) Average Latency" - Should be less than Database Reads (Attached) Average Latency when battery-backed write caching is utilized.
References: Exchange 2010 - https://technet.microsoft.com/en-us/library/ff367871(v=exchg.141).aspx & https://technet.microsoft.com/en-us/library/ff367896(v=exchg.141).aspx Exchange 2013 - https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx Exchange 2016 - Microsoft do not provide any documentation on performance counters at this time.
Exchange CAS Servers
"Processor(_Total)\% Processor Time" - Should be less than 75% on average. "Memory\% Committed Bytes In Use" - If this value is more than 80% (Exchange 2010) or 90% (Exchange 2013+) it is an indication that the system is likely under stress to provide more memory. "MSExchange RpcClientAccess\RPC Averaged Latency" - Should be less than 250ms "MSExchange RpcClientAccess\RPC Requests" - Should be less than 40
References: Exchange 2010 - https://technet.microsoft.com/en-us/library/ff367877(v=exchg.141).aspx & https://technet.microsoft.com/en-us/library/ff367896(v=exchg.141).aspx Exchange 2013 - https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx Exchange 2016 - Microsoft do not provide any documentation on performance counters at this time.
So i ran into this back in 2020, and mind you we were originally just backing up DB from a DAG. After about 2-3 months the backups for the mailboxes were awful. Taking days for 250 mailboxes, i found that Track stub movement option and removed it-
All my other check boxes are pretty similar other items highlighted above - but the removal the track stubs made the biggest impact. jobs for those mailboxes increased 600%
@jgeorges Great info!!
Outstanding post @jgeorges!
A few things to add, based on my more theoretical knowledge (vs real-world use and troubleshooting) and working with Dev over the years on questions from clients.
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.