The SDT data transfer was terminated on a request from the Job Manager

  • 11 August 2021
  • 5 replies
  • 1829 views

Badge +1

Hello All,

 

I am facing the same issue.  configured the file system backup for the server and after starting its full backup getting the error “encountered failure in receiving data [The SDT data transfer was terminated on a request from the Job Manager.]”

Applied the above suggested media management setting but no luck. 


 

17708 2e14  08/11 13:47:45 4370936 CPipelayer::SendPipelineBuffer() - Tail has reported error [94][The SDT data transfer was terminated on a request from the Job Manager.]. Cannot continue.

17708 2e14  08/11 13:47:45 4370936 [PIPELAYER  ] Error in flushing the current buffer.

17708 2e14  08/11 13:47:45 4370936 CVArchive::WriteBuffer() - Cannot send the buffer. Ret [268435460]

17708 2e14  08/11 13:47:45 4370936 CFileBackup::WriteBuffer(1683) - writeBuffer failed

17708 2e14  08/11 13:47:45 4370936 CFileBackup::HandleReadAndSendFileDataError(1497) - WriteBuffer failed

17708 2e14  08/11 13:47:45 4370936 CBackupBase::DoBackup(3689) - ReadAndSendFileData indicates FAIL_BACKUP

17708 2e14  08/11 13:47:45 4370936 CBackupBase::DoBackup(2608) - --- 8:29.499975 

17708 2e14  08/11 13:47:45 4370936 FsBackupTw::Run(510) - --- 8:29.502717 ObjectId=1, CollectFileName=C:\Program Files\Commvault\ContentStore\iDataAgent\JobResults\CV_JobResults\iDataAgent\FileSystemAgent\2\3585\NumColTot1.cvf

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::Run_WaitForThreadsToComplete(5364) - Wait for multiple objects signalled. Releasing one reservation used of object [1]

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::Run_WaitForThreadsToComplete_ReleaseStream(5412) - Worker thread reported error while processing collect file [C:\Program Files\Commvault\ContentStore\iDataAgent\JobResults\CV_JobResults\iDataAgent\FileSystemAgent\2\3585\NumColTot1.cvf]

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::PutInFileTheFailedFilesList(5662) - FailedFileCnt [0]

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::MarkCFAsProcessedInSubStoresReturned(10527) - mapSubStoreByCollect - size [0]

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::Run_WaitForThreadsToComplete_ReleaseStream(5496) - iLatestObjectSignalled=[1], iCurrentlyActiveStreams=[0]

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::Run_WaitForThreadsToComplete(5351) - --- 8:29.502192 

17708 5974  08/11 13:47:45 4370936 FsBackupCtlr::Run(3588) - iLatestObjectSignalled=[1], iMaxNumberOfThreads=[1], iCurrentlyAvailableStreams=[1], iCurrentlyActiveStreams=[0]

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4352) - +++ 

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4420) - Updating counts, FailedFiles=0, FailedFolders=0

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4480) - Leaving archive file marked invalid

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4485) - Closing pipeline

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4491) - Sending PL_LAST to tail.

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4495) - [4370936] backupState(1=good)=0, closePlArgs.backupState=0

17708 5974  08/11 13:47:45 4370936 CVArchive::ClosePipeline() - Resetting allocated stream

17708 5974  08/11 13:47:45 4370936 [PIPELAYER  ] Data Pipe Is Down

17708 5974  08/11 13:47:45 4370936 CCVAPipelayer::SendPerfCountersToDSBackup() - Failed to allocate Perf-counters command buffer

17708 5974  08/11 13:47:45 4370936 CCVAPipelayer::ClosePipeline() - About to destroy Data Mover

17708 5974  08/11 13:47:45 4370936 [PIPELAYER  ] Data Pipe Is Down

17708 5974  08/11 13:47:45 4370936 CCVAPipelayer::SendCommandToDSBackup() - Failed to allocate a command buffer

17708 5974  08/11 13:47:45 4370936 CCVAPipelayer::ClosePipeline() - Failure sending datamover destroy

17708 5974  08/11 13:47:45 4370936 CPipelayer::ShutdownPipeline() - stat- SDT [00000148929A00C0] [duration - 513 seconds]

17708 5974  08/11 13:47:45 4370936 CPipelayer::ShutdownPipeline() - pipeline has already been shutdown

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4503) - Backup: Errors in closing the pipeline.

17708 5974  08/11 13:47:45 4370936 CBackupBase::Close(4505) - Back from closePipeline


5 replies

Userlevel 7
Badge +23

Hi @G@rim@ !  I moved this to its own thread so we can better solve your issue (and the other thread didn’t resolve it for you).

Moved from:

Can you answer a few questions?

  1. What version is the Commserve as well as this client?  I have seen some issues where a mismatch causes this issue.
  2. Is there a firewall between the CS and Client? 
    1. Is the Media Agent the same as the CS?  If not, is there a firewall involved?

The error indicates that a connection was dropped and is usually indicative of a firewall issue or network issue.

We often notice setting up a dedicated consistent firewall within Commvault alleviates many network issues:

https://documentation.commvault.com/commvault/v11/article?p=7198.htm

Badge +1

@MichaelCapon I have checked the Job controller logs and i can see below errors in it. 

 

912  1d40  08/25 00:51:54 4407044 Control    JMASYNC sendControlMessageAsync network error while sending control message. Handle[81539], TimeOut[false],ErrorString [0x80070306:{ServiceCallBack(1241)/Couldn't connect to remote service. Handle (81539), Message (0x80070306:{ServiceCallBack(1210)} + {CCvIOCPServer::WorkerThread(1939)/W32.774.(One or more errors occurred while processing the request. (ERROR_ERRORS_ENCOUNTERED.774))-Connection closed.}), connection 

 

 

Badge +1

@Mike Struening Both Client and Commserve are running on the same version. and no firewall exist between client and MA. 

Also i have tried this setting but doesn’t worked for us. 

Userlevel 7
Badge +23

@G@rim@ , can you monitor the traffic between the 2 servers (with Wireshark or similar) to see if there are any drops?

The logging indicates network drops.  the One Way Firewall config helps by keeping a persistent pipe but if that is not working, it’s likely something deeper.

Userlevel 6
Badge +14

I would have expected the one-way persistent tunnel to resolve that error from occurring. - This error is usually caused by connection drop.
We may need to check clBackup, cvd and cvfwd logs (On the Client) leading up to the time of the error. 

 

Best Regards,

Michael

Reply