Solved

OneDrive user backup with large amount of data issue

  • 12 October 2021
  • 4 replies
  • 25 views

Userlevel 2
Badge +7

Hello Commvault Community, 

 

Today I come with the question of what might be causing the problem with the OneDrive backup job, but only for one user who has a large amount of data.

 

Users with little data are successfully backed up. 

 

This user was separated from a separate Sub-Client and a backup job was forced. It runs long and loops.

In the log on the Access Node (CvCloudBkpOneDrive.log) you can see for this backup job:


 

1) For a long time, for different files with different file extensions
 

14220 38b0 10/07 15:56:06 222703 OneDriveBackup :: RecordFailedItem (2090) - * Backup failed for (/OneDrive/xyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyz
14220 38b0 10/07 15:56:06 222703 OneDriveBackup :: RecordFailedItem (2091) - * [File download failure.]
14220 38b0 10/07 15:56:06 222703 OneDriveBackup :: RecordFailedItem (2090) - * Backup failed for (/OneDrive/xyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyz

 

2) After some time you can see an entry about the problem with Throttling:

4220 1 10/07 16:16:29 222703 ### callbackPermissionList - Throttling error, responseId (10524)
14220 1 10/07 16:16:29 222703 ### callbackPermissionList - Throttling error, responseId (10523)
14220 1 10/07 16:16:29 222703 ### - - * Wait (120) seconds because of throttling error in batch request
14220 1 10/07 16:18:59 222703 ### callbackPermissionList - Throttling error, responseId (11006)
14220 1 10/07 16:18:59 222703 ### callbackPermissionList - Throttling error, responseId (11010)
14220 1 10/07 16:18:59 222703 ### callbackPermissionList - Throttling error, responseId (11009)
14220 1 10/07 16:18:59 222703 ### callbackPermissionList - Throttling error, responseId (11008)
14220 1 10/07 16:18:59 222703 ### - - * Wait (120) seconds because of throttling error in batch request
14220 38b0 07/10 16:24:37 222703 OneDriveBackup :: StartBackup (2705) - * Start Backup for folders. mMapFileMetadata [14140]
14220 38b0 10/07 16:24:37 222703 OneDriveBackup :: UpdateProgress (1268) - * progress = 0.007072 (1/14140)
14220 1 10/07 16:24:37 222703 ### BatchGetPermissions - * Total elapsed Time for 14140 files: 0 Days 0 Hours 27 Minutes 30 Seconds 127 Milliseconds
14220 38b0 10/07 16:24:41 222703 OneDriveBackup :: StartBackup (2761) - * Download start for 13043 items
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - result: Amib.Threading.Internal.WorkItem + WorkItemResult
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - downloadsize: 6235919, requestindex: 4
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - result: Amib.Threading.Internal.WorkItem + WorkItemResult
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - downloadsize: 14312, requestindex: 5
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - result: Amib.Threading.Internal.WorkItem + WorkItemResult
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - downloadsize: 3260636, requestindex: 1
14220 1 10/07 16:25:06 222703 ### callbackDownloadForFileInfo_afterdownload - result: Amib.Threading.Internal.WorkItem + WorkItemResult


The CVD.log shows:

5284 9344 10/10 05:31:37 ###### [DM_MONITOR] Warning: DataMover for the jobs [222703] is in state [9] [DM_NETWORK]. State change time [Thu Oct 7 16:29:02 2021]
5284 48e0 10/10 05:37:31 222703 SdtTail :: logStats: Clnt [srv-an1], PId [13508], Cnt - BsyProcsg / Allocs [121/77080]; Time - Procsg / NwRecv / Total [220111/7051/227162] secs, Bytes recvd [38459688] RCId [316520]
5284 48e0 10/10 05:37:31 222703 SdtTailSrv :: MainLoop: Clnt [SDTPipe_srv-an1_srv-ma1_222703_1633609888_13508_20876_0000022188C782B0], PId [13508] has not recvd n / w sec. Set [1/1], RecvIdx [1/2], MxId [13148/4588], Id [9960], Last msg @ [Thu Oct 7 16:29:02 2021] RCId [316520]
5284 9344 10/10 05:41:37 ###### [DM_MONITOR] Warning: DataMover for the jobs [222703] is in state [9] [DM_NETWORK]. State change time [Thu Oct 7 16:29:02 2021]
5284 9344 10/10 05:51:37 ###### [DM_MONITOR] Warning: DataMover for the jobs [222703] is in state [9] [DM_NETWORK]. State change time [Thu Oct 7 16:29:02 2021]
5284 48e0 10/10 05:52:31 222703 SdtTail :: logStats: Clnt [srv-an1], PId [13508], Cnt - BsyProcsg / Allocs [121/77080]; Time - Procsg / NwRecv / Total [221011/7051/228062] secs, Bytes recvd [38459688] RCId [316520]


 

My question is, what could be the reason for the behavior that for small users OneDrive backup works fine, and for this one large user the backup job is not working.

 

When a job is stopped at Job Events, it indicates "Backup job (222703) has been converted to a full.

 

I started looking, and recommended to check if by any chance the situation will improve after setting up Network Throttling on the subclient level, maybe you will have an idea where to look for a solution.’

 

Thanks for help

Regards
Kamil

icon

Best answer by Scott Reynolds 15 October 2021, 21:10

View original

4 replies

Userlevel 7
Badge +21

Hey @Kamil , assuming this is OneDrive for Office 365, can you confirm how many Azure Apps you have configured?  You might be getting throttled on the Azure side.

Multiple Apps allow us to move to different apps as throttling gets close to occurring.

Userlevel 2
Badge +7

Hello @Mike Struening 

 

Thanks for quick reply. 

 

Customer have the minimum amount and is given in the documentation, i.e. 5 (azureapps.png)

 

Regards,

Kamil

Userlevel 7
Badge +21

@Kamil , how many users do you have total?  And how big are these larger ones?

I’m reaching out to some of our internal support team members to see what they can suggest.

Userlevel 2
Badge +2

@Kamil 

From the log cuts you posted it looks like it's possible this particular user has not only a large amount of data but possibly some very large files as well? The cuts are showing we are attempting to break up with large files into download chunks. The second section just shows we are waiting for the download to complete so the job stops reporting progress while this large download is occurring.

I would start by creating a few more azure apps to help with throttling. Another 5 for a total of 10 should be sufficient. Once that is complete run a backup of just this particular user without when the other backup is not running and see how it performs.

If you still have an issue open a support case and respond with the case number. It is possible we can provide a setting to skip the large files to see if the backup completes without them depending on the log review.

Reply