LPAR AIX, DB2 high CPU consumption

Adding to @Lukas_S . For the same set of backups below are the additional settings used, and commvault registry settings are not providing the expected results on the Client with reduction in CPU. Please share feedback and if this is to do with any issues within product or any other alternate suggestion to perform.

Server Compute: 

Entitled Capacity                          : 3.75
Online Virtual CPUs                        : 5
Online Memory                              : 131072 MB
Entitled Capacity of Pool                  : 1505

Commvault Parameters , CPU utilization and backup run time:

sSDTHeadMaxCPUUsage - 25%

Commvault Parameters: Streams-8, buffers-8, buffer size-4096,parallelism-8 : CPU utilization 90% avg, backup runtime: 1.5 hours

Commvault Parameters: Streams-5, buffers-5, buffer size-2048,parallelism-5 : CPU utilization 75% avg

backup runtime: 6 hours

Commvault Parameters , CPU utilization and backup run time:

sSDTHeadMaxCPUUsage - 25%

dnicevalue - 15

process_priority - 1

Commvault Parameters: Streams-8, buffers-8, buffer size-4096,parallelism-8 : CPU utilization 85% avg; Backup runtime: 2.20 minutes

Hello,

thank you both for your answers.

The Client is running on FR20, so it’s supposed to be running with LZO algorithm, but still it’ll be worth to check. Maybe compression, not deduplication is the clue. Because MediaAgent side dedup, still didn’t gave any expected results.

At the end, this seems to be the real issue, because performance peaks shown when backup software was changed:

“In virtualized environments (e.g.,LPAR, WPAR, Solaris Zones etc.,) where dedicated CPUs are not allocated, backup jobs may result in high CPU usage on production servers. The following measures can be taken to optimize the CPU usage”

Is there a way we can test other sets of commands being used during backup? Are devs working on new approach / engine regarding to LPAR? Whole virtualization is based on the using not dedicated cores, but we don’t see such CPU peaks during VMware or Hyper-V backups. 

regards,

Łukasz. 

Hello Damian,

actually agent was installed on FR20, so the LZO algorithm is in place. 

Correct, so we’d try going without compression at all with MA side dedup in place. 

Thank you. 

@Lukas_S , I’m going to see if we can get some of our support folks to chime in, though a support case might end up being best here.

I wasn’t able to get anything helpful quickly.  Create a support case and share the incident number here so I can track it accordingly 

Hello Mike, yes, but it will be long lasting case.

regards,

Łukasz.

Sharing the case resolution:

Experiencing high CPU in comparison to TSM backups for DB2.

Provided detailed analysis on what's using CPU methods to reduce this.

sSDTHeadMaxCPUUsage although set to 25% not having the desired effect:

DB2SBT log:
8454448 1 04/05 06:46:01 ####### SDT max. CPU thread count is [10] based on reg. value [25%], Procsr count [40]
8454448 1 04/05 06:46:01 ####### SdtBase::InitWrkPool: Initializing SDT head thread pool
8454448 1 04/05 06:46:01 ####### Max head thread count set to 40. CPU # = 40
8454448 1 04/05 06:46:01 ####### Threads per connection set to 20
8454448 1 04/05 06:46:01 ####### Initial max. threads set to 40

Logs indicate that we see 40 CPU's however the machine actually only has 5 virtual CPU's (lpar). We are performing calculation based on 40 CPU's meaning 24% of 40 = 10 and so 10 CPU threads used rather than expected 25% of 5 (1 or 2 rounded up).

This should be amended to specify threads rather than % of CPU - during session for testing we changed this to 2.

Noticed that deduplication is happening on client but the understanding was that this was happening on media agent:

DB2SBT log:
8126636 1 04/04 21:00:56 3472782 CPipelayer::InitiatePipeline signatureType [CV_SIGNATURE_SHA_512], signatureWhere [CV_CLIENTSIDE_DEDUP]

This is because (by default) storage policy has setting enabled to perform deduplication on clients.

That setting overrides the subclient setting to perform on Media agent as per the note in the subclient properties.

For testing purposes, disabled deduplication for the subclient to test - this will mimic deduplication not happening on the client.

The solution here could be to create a new storage policy and for clients where deduplication must happen on the media agent have the 'Enable Deduplication on Clients' setting disabled.

Encryption enabled for agent side (client). This will consume CPU cycles as well. For testing, changed this setting to 'Media Only'.

There are a couple of other factors:

Disabling Checksum (CRC) checking at the client side - Network CRC helps us detect corruption caused during network transfer but on some systems / processors this can consume alot of CPU cycles.

This can only be disabled at media agent level, see https://documentation.commvault.com/v11/expert/qscript/setMediaAgentProperty.html

It is possible to disable at client level however this would require an escalation to our development team to confirm CRC checking is actually the cause of high CPU usage.

Resource Control Groups - See https://documentation.commvault.com/commvault/v11_sp20/article?p=4954.htm another method to control / throttle CPU usage..
Target CPU usage of 40 - 50 % during their backups has now been achieved.

Hello,

do you know how we can check how nCOMPRESSIONSCHEME settings is set in a client?

Thanks

Lucio

I’ve never set the parameter “nCOMPRESSIONSCHEME” so I would like to understand what clients are using gzip (nCOMPRESSIONSCHEME=0) or lzo (nCOMPRESSIONSCHEME=1).

We are starting with a new HyperscaleX cluster, and that would be the right moment to have some tests and use the best setting.

I think that it would be useful to understand also the compression method used for an already completed backup job.

Lucio

Thanks Mike,

I’m going to check all the clients settings.

I’ve just took a DB2 backup test with nCOMPRESSIONSCHEME=1 on an AIX LPAR, and the CPU utilization was about 50% less, with improved throughput, even considering that it was a first backup, with all new data written.

Now I’ve to evaluate if the dedup ratio is comparable and acceptable, but from the CPU point of view there is a big difference.

I perfectly understand the decision not to implement this change automatically on all clients, but in my opinion this “feature” should be advertised better, considering the benefits.

On IBM Power9 with AIX 7.2 there would also be an hardware accellerated version of gzip, that performs 10x better than the original one (https://community.ibm.com/community/user/power/blogs/brian-veale1/2020/11/09/power9-gzip-data-acceleration-with-ibm-aix).

Best Regards

Lucio

I agree, I ran into the same issue in 2018, maybe just as a footnote to the documentation. At the time we also went through the process of contacting support, so the documentation update will likely save you guys some internal cycles.