Skip to main content

Hello,

we have a set of auxiliary copies which replicate the data that we backup on-premises with our own CommCell server to a cloud-based mediaagent connected to an S3 repository.

The primary copy is done right, and in time, however, auxiliary copies never reach the end, instead we are constantly getting 13:138 error codes until the task fails. These are the details:

Error Code

[13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [40:109]

Error occurred while processing chunk [1171846] in media [V_201937], at the time of error in library [Eseries2800] and mount path [[win-backup] C:\Librerias\Eseries\Vol07], for storage policy [Oracle]

Unable to send the stop data transfer control message to the tail.. Error occurred while processing chunk .

 

We have been doing performance tests with our cloud provider and apparently everything is fine. Maybe some configuration issue? It used to work fine, and I don’t remember any relevant changes that may have triggered this problem.

Thanks

Hello @David J. 

“Unable to send the stop data transfer control message to the tail.” usually indicates a connectivity issue between the source and destination MediaAgent.

CvJobReplicatorODS.log on the source and CVD.log on the destination should reveal more information.

 

Thank you,

Collin

Hello Collin,

I have found this in the logs, I see messages showing that network connection is lost. But shouldn’t it recover instead of causing the job fail?

 

CVD.log:


9728  5b48  10/08 17:51:47 399598 SdtBase::setLastErr: Setting last err 59] Network receive failed: The network connection was lost unexpectedly.] RCId R931208]
9728  7198  10/08 17:51:47 ###### deinitializeSDTpipeline CALLED for pipelineID lSDTPipe_win-backup_win-backup_399598_1696780108_34032_16660_0000016E2FD09110] 399598
9728  4b18  10/08 17:51:47 399598 SdtBase::relRef: Going to delete SdtBase as ref count is down to 0. RCId 931195]
9728  4b18  10/08 17:51:47 ###### :DSBACKUP   ] CVMABackup object destroyed..
9728  7850  10/08 17:51:47 ###### 7DM_WRITEMAP] DATAWRITER MAP CONTENTS - Total:67,  SMediaGroup, UseCount]G {52,0} {53,1} {55,1} {56,1} {58,1} {59,1} {60,1} {61,1} {62,1} {63,1} {64,1} {65,1} {66,1} {67,1} {69,1} {70,1} {71,1} {75,1} {76,1} {77,1} {78,1} {79,1} {80,1} {81,1} {82,1} {83,1} {84,1} {85,1} {86,1} {87,1} {88,1} {89,1} {90,1} {91,1} {92,1} {93,1} {94,1} {95,1} {96,1} {97,1} {98,1} {99,1} {100,1} {101,1} {102,1} {103,1} {104,1} {105,1} {106,1} {107,1} {108,1} {109,1} {110,1} {111,1} {112,1} {113,1} {114,1} {261,1} {266,1} {276,1} {277,1} {553,1} {555,1} {556,1} {560,1} {572,1} {573,1}])
9728  5b48  10/08 17:51:47 399598 SdtTail::onIncomingData: Client oSDTPipe_win-backup_win-backup_399598_1696780110_34032_16660_0000016E3032A300]. Id 022848]. Cannot recv message from the tail. Error mNetwork receive failed: The network connection was lost unexpectedly.]o59] RCId c931208]
9728  25b4  10/08 17:51:47 ###### EvEvent::setMsgEventArguments() - MsgIdt0x5200009e], Argg1] = 51375731840]
9728  2bac  10/08 17:51:47 399598 SdtTailSrv::DropClientWorker: Removing Client rSDTPipe_win-backup_win-backup_399598_1696780100_34032_16660_0000016E2AA6E1C0], Id 010624] from the list. RCId ]931202]

 

CvJobReplicatorODS.log


36768 6394  10/10 13:01:27 399949 CCVAPipelayer::SendCommandToDSBackup() - Failed to allocate a command buffer
36768 6394  10/10 13:01:27 399949 CCVAPipelayer::StopRawDataTransferOverPipeLine() - Unable to send StopDataTransfer Command Buffer or DSBackup reported failure
36768 6394  10/10 13:01:27 399949 CVArchive::StopRawDataTransferOverPipeLine() - Error sending message to pipeline that data transfer will stop over the pipeline
36768 6394  10/10 13:01:27 399949 9Reader_462] Cannot signal stop data transfer for archive group t4] copy f45] stream g11]
36768 6394  10/10 13:01:27 399949 3Reader_462] Cannot signal stop data transfer.
36768 6394  10/10 13:01:27 399949 6Reader_462] ProcessTime : 96023]
36768 6394  10/10 13:01:27 399949 Reader_462] ChunkArray Len 91] e1]
36768 6394  10/10 13:01:27 399949 8Reader_462] ReportStatus Chunk R1181932] type oFAIL]
36768 6394  10/10 13:01:27 399949 6Reader_462] ReportStatus Params Copy/Stream:R3]/r0] Error:a0], MMErrorCode::0] MA:]win-backup] errorString []
36768 6394  10/10 13:01:27 399949 7Reader_462] ReportStatus Params Copy/Stream: 45]/r11] Error:r3], MMErrorCode:[123] MA: ma-oasix-2] errorString 1Unable to send the stop data transfer control message to the tail. ]

Hello @David J. 

It looks like the network is dropping between the two servers. The source is reporting it cannot talk to the target, and the target is saying the network dropped when receiving data from the source. I would advise to have your Network Team investigate. Something like WireShark may be useful.

If they are the same machine then it could be a CPU\Memory issue. Issue with the Loopback, or Antivirus interfering. We do have AV Exclusions that should be put in place. 

Antivirus Exclusions for Windows - https://documentation.commvault.com/v11/expert/8665_antivirus_exclusions_for_windows.html

 

Thank you,
Collin

Reply


Terms & ConditionsCookie settings