Skip to main content
commvault.com commvault.com
Question

Problems with auxiliary copy

  • October 9, 2023
  • 3 replies
  • 658 views
D
Novice
Forum|alt.badge.img

Hello,

we have a set of auxiliary copies which replicate the data that we backup on-premises with our own CommCell server to a cloud-based mediaagent connected to an S3 repository.

The primary copy is done right, and in time, however, auxiliary copies never reach the end, instead we are constantly getting 13:138 error codes until the task fails. These are the details:

Error Code

[13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [13:138] [40:109]

Error occurred while processing chunk [1171846] in media [V_201937], at the time of error in library [Eseries2800] and mount path [[win-backup] C:\Librerias\Eseries\Vol07], for storage policy [Oracle]

Unable to send the stop data transfer control message to the tail.. Error occurred while processing chunk .

 

We have been doing performance tests with our cloud provider and apparently everything is fine. Maybe some configuration issue? It used to work fine, and I don’t remember any relevant changes that may have triggered this problem.

Thanks

    3 replies

    C
    Vaulter
    Forum|alt.badge.img+14

    Hello @David J. 

    “Unable to send the stop data transfer control message to the tail.” usually indicates a connectivity issue between the source and destination MediaAgent.

    CvJobReplicatorODS.log on the source and CVD.log on the destination should reveal more information.

     

    Thank you,

    Collin

    D
    Novice
    Forum|alt.badge.img
    • David J.Novice
    • Author
    • Novice
    • October 11, 2023

    Hello Collin,

    I have found this in the logs, I see messages showing that network connection is lost. But shouldn’t it recover instead of causing the job fail?

     

    CVD.log:


    9728  5b48  10/08 17:51:47 399598 SdtBase::setLastErr: Setting last err [59][Network receive failed: The network connection was lost unexpectedly.] RCId [931208]
    9728  7198  10/08 17:51:47 ###### deinitializeSDTpipeline CALLED for pipelineID [SDTPipe_win-backup_win-backup_399598_1696780108_34032_16660_0000016E2FD09110] 399598
    9728  4b18  10/08 17:51:47 399598 SdtBase::relRef: Going to delete SdtBase as ref count is down to 0. RCId [931195]
    9728  4b18  10/08 17:51:47 ###### [DSBACKUP   ] CVMABackup object destroyed..
    9728  7850  10/08 17:51:47 ###### [DM_WRITEMAP] DATAWRITER MAP CONTENTS - Total:67,  [MediaGroup, UseCount][ {52,0} {53,1} {55,1} {56,1} {58,1} {59,1} {60,1} {61,1} {62,1} {63,1} {64,1} {65,1} {66,1} {67,1} {69,1} {70,1} {71,1} {75,1} {76,1} {77,1} {78,1} {79,1} {80,1} {81,1} {82,1} {83,1} {84,1} {85,1} {86,1} {87,1} {88,1} {89,1} {90,1} {91,1} {92,1} {93,1} {94,1} {95,1} {96,1} {97,1} {98,1} {99,1} {100,1} {101,1} {102,1} {103,1} {104,1} {105,1} {106,1} {107,1} {108,1} {109,1} {110,1} {111,1} {112,1} {113,1} {114,1} {261,1} {266,1} {276,1} {277,1} {553,1} {555,1} {556,1} {560,1} {572,1} {573,1}])
    9728  5b48  10/08 17:51:47 399598 SdtTail::onIncomingData: Client [SDTPipe_win-backup_win-backup_399598_1696780110_34032_16660_0000016E3032A300]. Id [22848]. Cannot recv message from the tail. Error [Network receive failed: The network connection was lost unexpectedly.][59] RCId [931208]
    9728  25b4  10/08 17:51:47 ###### EvEvent::setMsgEventArguments() - MsgId[0x5200009e], Arg[1] = [1375731840]
    9728  2bac  10/08 17:51:47 399598 SdtTailSrv::DropClientWorker: Removing Client [SDTPipe_win-backup_win-backup_399598_1696780100_34032_16660_0000016E2AA6E1C0], Id [10624] from the list. RCId [931202]

     

    CvJobReplicatorODS.log


    36768 6394  10/10 13:01:27 399949 CCVAPipelayer::SendCommandToDSBackup() - Failed to allocate a command buffer
    36768 6394  10/10 13:01:27 399949 CCVAPipelayer::StopRawDataTransferOverPipeLine() - Unable to send StopDataTransfer Command Buffer or DSBackup reported failure
    36768 6394  10/10 13:01:27 399949 CVArchive::StopRawDataTransferOverPipeLine() - Error sending message to pipeline that data transfer will stop over the pipeline
    36768 6394  10/10 13:01:27 399949 [Reader_462] Cannot signal stop data transfer for archive group [4] copy [45] stream [11]
    36768 6394  10/10 13:01:27 399949 [Reader_462] Cannot signal stop data transfer.
    36768 6394  10/10 13:01:27 399949 [Reader_462] ProcessTime : [6023]
    36768 6394  10/10 13:01:27 399949 [Reader_462] ChunkArray Len [1] [1]
    36768 6394  10/10 13:01:27 399949 [Reader_462] ReportStatus Chunk [1181932] type [FAIL]
    36768 6394  10/10 13:01:27 399949 [Reader_462] ReportStatus Params Copy/Stream:[3]/[0] Error:[0], MMErrorCode:[0] MA:[win-backup] errorString []
    36768 6394  10/10 13:01:27 399949 [Reader_462] ReportStatus Params Copy/Stream:[45]/[11] Error:[3], MMErrorCode:[123] MA:[ma-oasix-2] errorString [Unable to send the stop data transfer control message to the tail. ]

    C
    Vaulter
    Forum|alt.badge.img+14

    Hello @David J. 

    It looks like the network is dropping between the two servers. The source is reporting it cannot talk to the target, and the target is saying the network dropped when receiving data from the source. I would advise to have your Network Team investigate. Something like WireShark may be useful.

    If they are the same machine then it could be a CPU\Memory issue. Issue with the Loopback, or Antivirus interfering. We do have AV Exclusions that should be put in place. 

    Antivirus Exclusions for Windows - https://documentation.commvault.com/v11/expert/8665_antivirus_exclusions_for_windows.html

     

    Thank you,
    Collin

    Terms & ConditionsCookie settingsAccessibility statement