Skip to main content
Question

Slow performance while restoring a Full VM


Forum|alt.badge.img+6

Hi All,

We have a VM of 1.5TB, iDA: Virtual Server, Vendor: VMware.

While we are restoring the VM with out of place to same destination as source, we are getting low throughput i.e., ~29GB/hr where our network is 10GB/s.

This VM is a Windows 2019 machine with latest VMware Tools installed and we have a VSA in same cluster of VM & a physical Linux server as a Media Agent in the same LAN.

While backup, it was taking HotAdd mode and the throughput was 137GB/hr and elapsed time was about 6hrs for backup, but for restoring it taking NBD mode and throughput is too low & time taking restore was more than 24hrs.

Can someone please let me know, is this restore was slow because the speed of writes on the datastores & disks was low? and also no.of files can affect the speed?

If yes, how can we increase the speed of this restore?

If not, how can we find the cause of the slowness, any logs to check or any tool to use like CvDiskPerf tool or CvNetworkTestTool?

Is there any additional setting that can help me to do faster restore?

 

Below are my findings as per my knowledge,

Job log:

10460 554   07/07 20:11:49 4682894 VSRstWorker::UpdateRestoreProgress() - VM [Test_VMRestore] Restored 694488 of 757508 extents 91%

10460 554   07/07 20:10:13 4682894 stat- ID [writedisk - Test_RS_2_4.vmdk], Bytes [497151901696], Time [51718.153326] Sec(s), Average Speed [9.167400] MB/Sec
10460 554   07/07 20:10:45 4682894 stat- ID [readmedia], Bytes [722762550223], Time [18309.236606] Sec(s), Average Speed [37.646577] MB/Sec
10460 554   07/07 20:10:45 4682894 stat- ID [Datastore Write [VxRail-Datastore01]], Bytes [727869030400], Time [82336.959808] Sec(s), Average Speed [8.430600] MB/Sec

 

CVD.log of Media Agent:

2593503 b8cb7  07/06 21:37:34 4682894 [FSRESTHEAD ] 1668180256-# SendCtrlBuf: Sending Ctrl Msg 0x14C
2593503 b8cb8  07/06 21:45:44 4682894 [FSRESTHEAD ] 1668180256-# Stats: AppTy 106, AF 6, Seeks 3536/3553/68983, WrkItms 56952, MxRdr (Bsy-Tot: 1-3, Pnd 0), MsgsRcvd 2185, QLen 2046, LstMsg CLOSE_ARCHIVE
2593503 b8cb8  07/06 21:45:44 4682894 [FSRESTHEAD ] 1668180256-# Stats: [AllocBuf Time] Exp Avg [0.00], Total Avg [0.02], Total Time [1640.94], Total Count [96120]
2593503 b8cb8  07/06 21:45:44 4682894 [FSRESTHEAD ] 1668180256-# Stats: [AllocBuf Speed] Curr Avg [5.02] MB/Sec, Bytes [3163522296]; Total Avg [3.66] MB/Sec, Bytes [6295475520]
2593503 b8cb8  07/06 21:45:44 4682894 [FSRESTHEAD ] 1668180256-# Stats: [MediaRead Time] Exp Avg [0.00], Total Avg [0.00], Total Time [3.20], Total Count [96120]
2593503 b8cb8  07/06 21:45:44 4682894 [FSRESTHEAD ] 1668180256-# Stats: [MediaRead Speed] Curr Avg [731.07] MB/Sec, Bytes [940240258]; Total Avg [581.90] MB/Sec, Bytes [1952738661]

 

Thanks in advance.

Warm Regards,

Sravan Kondeti

2 replies

Forum|alt.badge.img+15

Hello @backup 

Thanks for the question and all the detail. The CVPerfMgrRestore.log is going to be the best answer for you. This log will write a summary of a closed stream and all of its performance metrics. Can you please look on the media agent that wrote the data for the log line relating to your JobID and post it here? 

 

Kind regards

Albert Williams


Forum|alt.badge.img+14

Hi @backup ,

What speed are you having using Hotadd?

What NIC do you have on that VSA?

Can you check this KB:

https://kb.commvault.com/article/60399

Can you run TestVMInfo?

Best Regards,

Sebastien


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings