Question

Poor backup performance for MongoDB

  • 29 September 2022
  • 6 replies
  • 108 views

Userlevel 2
Badge +7

Hi team,

 

I’m conducting a PoC for MongoDB sharded cluster.

Customer runs two shards and each shard has 3nodes. (1 primary and 2 secondary)

We’ve installed plug-in on all 6 nodes and all configuration went well.

 

For 1st shard, 

we performed the initial full backup without any option.

For second shard,

we performed the initial full backup with some option for better performance.

...

 

Backup performance is not bad in the beginning. (100-150MB/sec)

Backup size of shard is 5TB and should be able to complete the job within 20hrs.

So performance need to be higher than 80-100MB/sec at least.

 

Backup server and Mongodb nodes spec is enough and resources are always available.

But backup peformance is gradually degraded as time goes by and comes to 5-6MB/sec.

We’ve never completed the initial backup due to this unexpected performance degradation.

Mongodb data is on SSD and connected with 10Gb network for backup.

 

CVLT version is 11.28 and also tried 11.26 but the same result.

I’m sure that installation and settings for backup is perfect and LVM snap works well.

 

Have you guys experienced this symptom?

Could you give me an input need to look into again?

 

Thanks.


6 replies

Userlevel 4
Badge +12

Hi @Hongmo Kim 

 

Please review the CvPerfMgr.log on the media agent for the backup job to find the bottleneck.

 

Regards

Userlevel 5
Badge +11

@Hongmo Kim 

 

Please remove those additional settings from your post, they both require engineering escalation before usage. 

 

Thanks 

Userlevel 2
Badge +7

@Jordan 

I should have been cautious …  I’m sorry.

Userlevel 7
Badge +19

So did removing the additional settings made any difference? 

Userlevel 2
Badge +7

@Onno van den Berg 

also tried without additional settings but same symptom.

Additionally tried it with no compression and deduplication but no difference.

 

Badge

Hi! Do you have solution for this issue? I have same with 6TB MongoDB cluster. The bandwidth of backup is degraded after 1TB

Reply