commvault.com commvault.com
Solved

CV/Netapp A700 config permits only 36 concurrent sessions for backup copy jobs to a FAS8700: Is there a way to increase this ?

  • 23 October 2023
  • 3 replies
  • 101 views
M
Rank
Userlevel 1
Badge +5

 

 

CV is configured with a cluster ndmp client and the nodes for the cluster are added in the ndmp client configuration. There are 6 physical nodes in the cluster. Source netapp is an a700 and the destination netapp is a FAS8700.

Issue:

   For a backup copy job hundreds of jobs running but only a max of 36 transfer data. Hundreds of jobs trying to make a connection to the filer, but they can’t.  These jobs are each grabbing a stream but only 36 at a time are moving data. Increasing streams helps but then more backup copy jobs that can run launch and consume streams.

There is a snap primary, storagegrid backup copy, vault replica copy,  and streaming data protection jobs that are very behind because of the supposed concurrent session limit (36 ?) from the source a700: streams keep being eaten up by backup copy jobs, but only 36 move data concurrently, thereby causing jobs for other copies to not run including the snap primary source copy for the storagegrid backup copy jobs. 

Misc.

From the customer netapp architect: “CV has cluster aware functionality; is backing up volumes through the svm’s that are on that cluster node. The customer netapp architect says the cluster can hand out only 36 simultaneous ndmp sessions. There is a netapp hard limit for 36 sessions if you go through that cluster. Can’t set it higher.”

Idk if he actually knows what he is talking about or not.  

My Questions:

Is it supposed to be handing out more sessions through these svms ? Could the configuration be done differently ?  Could each svm do 36 simultaneous ndmp sessions ? Is the customer architect correct and somehow a netapp a700 will only allow 36 simultaneous backup copy jobs from it ?

icon

Best answer by Swati Himmatramka 27 October 2023, 10:05

View original

3 replies

S
Rank
Badge +3

Hello Mzator,

Maximum number of NDMP sessions that can be established simultaneously on storage systems depends on the system memory of a storage system.

As per the Netapp doc, we could see below values:

System memory of a storage system Maximum number of NDMP sessions

Less than 16 GB

8

Greater than or equal to 16 GB but less than 24 GB

20

Greater than or equal to 24 GB

36

https://docs.netapp.com/us-en/ontap/tape-backup/scalability-limits-ndmp-sessions-reference.html

If the session limit is reached it will affect the jobs hence, regarding the session concern we would suggest checking with your Netapp team and check the configuration requirement for NDMP session. 

Commvault software has no way to increase the NDMP sessions.

Architect seems to be fine however, it is due to the session limit on the Filer end. 

Also, you need to make sure the sessions are terminating correctly on the filer and no stale entry.

You can try dividing the content into multiple subclients so that the sessions to the Filer gets completed faster and other NDMP Backups can get session from the Filer.

Or you can divide the backup schedule in different time so that none of the NDMP Backup overlaps with each other.

M
Rank
Userlevel 1
Badge +5

Swati thank you, it is as I thought. Can you speak to what is seen when sessions are not terminating correctly on the Filer ? What is seen in CV and in the netapp ?

S
Rank
Badge +3

In CV, job will throw an error when the maximum contect is reached “Maximum number of backup contexts reached”.

It needs to be reviewed on the Filer end if there is any stale session which is not cleared and can be deleted. For more information on this you can out reach to the Netapp Vendor. 

Note:  You can retrieve and delete NDMP session details for a specific SVM or node. This resource type was introduced with ONTAP 9.7.

Ideally Data Management Application (DMA) is the backup application i.e Commvault here, and is responsible for creating ndmp sessions, and for closing ndmp sessions as soon they are no longer required.

In case if it fails to cleanup ndmp sessions then it can cleared from filer to free up the resources using "system services ndmp kill " command.

In the logs it could be similar to “NDMP_BRE_DELETE_CONTEXT: got reply error [0x20500501=NDMP_BRE_CONTEXT_BUSY_ERR]” error → if it fails to delete the context.

 

Reply


Terms & ConditionsCookie settings