Solved

SharePoint Document Backup Hanging

  • 17 March 2021
  • 9 replies
  • 716 views

Userlevel 3
Badge +7

Hi,

I’m having an issue with Sharepoint Document backup and am wondering if anybody else had this issue before. The job will start and proceed and eventually hang.

In the logs on the sharepoint client, I find two interesting things:

CVSPBackup.log

7160  32cc  03/17 10:22:37 622711 CvThreadPool::waitForAllThreads() - Thread Pool [SharePointBackupThreadPool]. Could not finish all work in the time limit of [300] seconds. The worker list count is [0] and busy thread count is [1] and busy thread list is [0xdc0]
7160 19d8 03/17 10:24:03 622711 TPool [SharePointBackupThreadPool]. Ser# [0] Tot [1], Pend [1], Comp [0], Max Par [0], Time (Serial) [0.000000]s, Time (Parallel) [0.000000]s, Wait [0.000000]s
7160 32cc 03/17 10:27:38 622711 CvThreadPool::waitForAllThreads() - Thread Pool [SharePointBackupThreadPool]. Could not finish all work in the time limit of [300] seconds. The worker list count is [0] and busy thread count is [1] and busy thread list is [0xdc0]
7160 19d8 03/17 10:29:03 622711 TPool [SharePointBackupThreadPool]. Ser# [0] Tot [1], Pend [1], Comp [0], Max Par [0], Time (Serial) [0.000000]s, Time (Parallel) [0.000000]s, Wait [0.000000]s
7160 32cc 03/17 10:32:39 622711 CvThreadPool::waitForAllThreads() - Thread Pool [SharePointBackupThreadPool]. Could not finish all work in the time limit of [300] seconds. The worker list count is [0] and busy thread count is [1] and busy thread list is [0xdc0]
7160 19d8 03/17 10:34:03 622711 TPool [SharePointBackupThreadPool]. Ser# [0] Tot [1], Pend [1], Comp [0], Max Par [0], Time (Serial) [0.000000]s, Time (Parallel) [0.000000]s, Wait [0.000000]s
7160 32cc 03/17 10:37:40 622711 CvThreadPool::waitForAllThreads() - Thread Pool [SharePointBackupThreadPool]. Could not finish all work in the time limit of [300] seconds. The worker list count is [0] and busy thread count is [1] and busy thread list is [0xdc0]
7160 32cc 03/17 10:37:40 622711 CVSPMultiStreamBackup::WaitOnBackupThreadsToComplete() - Sending Keep Alive message for Stream {1}
7160 32cc 03/17 10:37:40 622711 CCVAPipelayer::SendCommandBuffer() - Sent Command 261 to MediaAgent, Waiting for Response...
7160 32cc 03/17 10:37:40 622711 [PIPELAYER ] Tail response = Keeping Head/Tail Socket Alive Status = 0 dwCode = 0 MsgType = 16 MsgSubType = 261 EventError = 0
7160 32cc 03/17 10:37:40 622711 CVSPLibNamespace::CVSPArchive::SendKeepAliveMessage() - Successfully sent PL_KEEP_ALIVE message

CCSDb.log

10880 3524  03/17 10:49:43 ### CCS_SupportedPackages::isCCSDbOperationAllowed() - Invalid Package NOT supported by CCS for PackageId[101]
10880 3524 03/17 10:49:43 ### ::CCSDbReloadCheckThread() - Configuration has been changed and not supported by CCSDb and it is being disabled. cfgState[0] disabled[1]
10880 1e0c 03/17 10:54:43 ### CCS_SupportedPackages::isCCSDbOperationAllowed() - Invalid Package NOT supported by CCS for PackageId[101]
10880 1e0c 03/17 10:54:43 ### ::CCSDbReloadCheckThread() - Configuration has been changed and not supported by CCSDb and it is being disabled. cfgState[0] disabled[1]
10880 e80 03/17 10:59:43 ### CCS_SupportedPackages::isCCSDbOperationAllowed() - Invalid Package NOT supported by CCS for PackageId[101]
10880 e80 03/17 10:59:43 ### ::CCSDbReloadCheckThread() - Configuration has been changed and not supported by CCSDb and it is being disabled. cfgState[0] disabled[1]
10880 3608 03/17 11:04:43 ### CCS_SupportedPackages::isCCSDbOperationAllowed() - Invalid Package NOT supported by CCS for PackageId[101]
10880 3608 03/17 11:04:43 ### ::CCSDbReloadCheckThread() - Configuration has been changed and not supported by CCSDb and it is being disabled. cfgState[0] disabled[1]
10880 3928 03/17 11:09:43 ### CCS_SupportedPackages::isCCSDbOperationAllowed() - Invalid Package NOT supported by CCS for PackageId[101]
10880 3928 03/17 11:09:43 ### ::CCSDbReloadCheckThread() - Configuration has been changed and not supported by CCSDb and it is being disabled. cfgState[0] disabled[1]

From the CVSPBackup log I gather nothing is going through and the pipeline is kept open by sending keep alives. This repeats itself for hours.

From the CCSDB log, I gathed some package is invalid and therefore disabled. However, I’m not sure what package this is. Is this the sharepoint agent?

I’m wondering if reinstalling the agent with a fresh package, instead of a package that was upgraded over and over again, will fix this.

Has anybody seen this issue before?

 

Kind regards,

Jeremy

 

 

icon

Best answer by Jeremy 28 March 2021, 10:54

View original

9 replies

Userlevel 7
Badge +15

Hi @Jeremy 

Thanks for the question, we need a little more info here to understand what’s happening.

What version, service pack and hotfix pack level are you running?

Are you running with Catalog ACL or All Versions enabled?

Thanks,

Stuart

Userlevel 3
Badge +7

Hi @Stuart Painter ,

Thanks for your reply.

We are running 11.20.36.

We are running without Catalog ACL, and All Versions is enabled on the schedule.

Kind regards,

Jeremy

Userlevel 7
Badge +15

Hi Jeremy

Thank you for the info, after a discussion within the support team here, this will need a detailed investigation, so please would you raise a support ticket so we can have a closer look.

If you don’t mind IM me the ticket number so I can track and post the solution here to share with the community?

Thanks,

Stuart

Userlevel 3
Badge +7

Hi Stuart,

No worries, I will raise a support ticket.

I’ll let you know the case number in a bit.

Kind regards,

Jeremy

Userlevel 7
Badge +15

Thanks Jeremy, I’ll track the support case.

Thanks,

Stuart

Userlevel 5
Badge +10

How large is your SharePoint Online site? I have seen a single site in the Terabytes with millions of files written to the access node that bogged down NTFS?

Userlevel 3
Badge +7

Hi @Anthony.Hodges

Thanks for your message.

We have two and they’re only about 20-25GB application size, and about 800 000 - 1 000 000 files.

I’m working with Support and we found one site that was causing the backup to hang. After isolating this site in its own subclient, the backups completed successfully.

We’re now trying to figure out what’s the issue with the particular site. 

Thanks.

Jeremy

Userlevel 3
Badge +7

Hi,

The particular site which was causing the Document backup to hang in Commvault, also caused a normal export via the native sharepoint powershell commands to hang. We agreed this issue is not with Commvault, but with the Sharepoint configuration. We advised to contact Microsoft in order to troubleshoot the issue.

Workaround is to exclude this particular site from the default subclient in its own subclient or filter it from the default subclient.

Kind regards,

Jeremy

Userlevel 7
Badge +15

Hi Jeremy

Thank you for the update, I hope you get the underlying Sharepoint issue resolved with Microsoft.

Thanks,

Stuart

 

Reply