Solved

Description: Error occurred while processing chunk - error Code: [13:138]


Userlevel 4
Badge +13

Hi all!

could you advice me how to troubleshoot following types of error:

 

Error Code: [13:138] Description: Error occurred while processing chunk [xxx] in media [xxx], at the time of error in library [disklib01] and mount path [[xxx] /srv/commvault/disklib01/xxx], for storage policy [XXX] copy [Xxx] MediaAgent [svma1]: Backup Job [xxx]. Unable to setup the copy pipeline. Please check connectivity between Source MA [svma1] and Destination MA [svma1].

 

At a glance, it seems that it is not possible for CV to process chunk from the (index?)/disk library...However, the issue is connected with storage policy copy, that moves data from the disk library to the tape library (secondary copy). The main problem for us is that it is not possible to copy data to the tapes. Therefore, it may say Unable to setup the copy pipeline. The media agent is one server/device, that communicates with both disk and tape library. Lastly, the files in the related directories dont seem to be corrupted...

Any suggestions and ideas will be appreciated!

 

 

icon

Best answer by Mike Struening RETIRED 2 September 2022, 21:09

View original

21 replies

Userlevel 7
Badge +23

@drPhil , based on the error, there’s a communication issue:

between Source MA [svma1] and Destination MA [svma1].

Can you check the connection reliability between those 2?

Userlevel 4
Badge +13

@drPhil , based on the error, there’s a communication issue:

between Source MA [svma1] and Destination MA [svma1].

Can you check the connection reliability between those 2?

Hi @Mike Struening, I hope you are doing well 😀

The thing is that there is only one Media Agent. Therefore, it looks a little bit incomprehensible.

Badge +2

What version of CV are you on? We started having these same errors after upgrading to 11.28.8.

We have the same error message intermittently occur where the media agent  can’t communicate with itself.

Unable to setup the copy pipeline. Please check connectivity between Source MA [ma004] and Destination MA [ma004].

We do have an active ticket with support.

Userlevel 7
Badge +23

@drPhil , I must be blind 🤣

Apologies for missing the obvious!

I concur with @RyanOJD .  If you are in the same boat version wise, create a support case.

If you can both share the incident number, I can track them.

Badge +2

@drPhil , I must be blind 🤣

Apologies for missing the obvious!

I concur with @RyanOJD .  If you are in the same boat version wise, create a support case.

If you can both share the incident number, I can track them.

Incident ID: 220712-540. 

Userlevel 7
Badge +23

Thank you!!

Badge +3

I’m seeing the same error messages as well.

Badge +2

After upgrading from 11.26.13 to 11.28.8 we began experiencing frequent errors on aux copy jobs, at times multiple errors per hour.

CV support escalated our support ticket to development. We had an existing additional setting on our HyperScale Media agents. The setting was sPipelineMode and it was set to AB:P

CV Development requested that we try removing this setting. After removing this setting the chunk errors with code 13:138 stopped occurring as did two other errors. The other two errors were timeout errors code 13:85 and unknown buffer type [0] received code 7:42. 

Userlevel 4
Badge +13

Hi all,

after investigating we have found out that there is an issue with the tape drive, that shows some errors on the management panel. I think, the error posted above (Error while processing chunk) is not related with the new version of Commvault since we have 11.24.48.

Our next step will be the most likely to obtain new tape drive. However, is it possible to change Global Secondary Policy Copy in the Policy Copy setting tab (it is grey out)? Now, we would need to use only one tape drive, but it is necessary to let the Policy Copy to choose different scratch pool...And Global Secondary Policy Copy can have only one scratch pool defined…..

Userlevel 5
Badge +13

Hello @drPhil 

You cannot re-associate a copy to a different Global Secondary Copy once its been associated.

You can put the issue drive into maintenance so that we won’t try to use it and you should be able to add the second scratch pool to the data paths of the Global Secondary Policy. 

Modifying the Properties of the Global Secondary Primary Copy Policy - https://documentation.commvault.com/11.26/expert/11653_modifying_properties_of_global_secondary_primary_copy_policy.html

 

Thank you,
Collin

Badge

Hello @Collin Harper 

We have seen this very error, “13:138 Error Occurred while processing chunk[...]”

[EDIT] Adding one more error message seen

“Failed to mount the disk media in library […] with mount path […] on mediaagent […]”

except this is during a job that backs up to an s3 bucket. 

Unfortunately I dont have access to the CV application myself, this is what I know of the problem and trying to see how can it be helped.

I dont know if there are tape libraries in the picture at the moment.

Is there anything you can point me to?

Thank you in advance!

Userlevel 2
Badge +7

Same issue happening now to me..it is for Tape library copy. Is the reason the drive need replace / faulty? is that a root cause (CRC error)? Error: Unable to setup the copy pipeline. Please check connectivity between Source MA [xxx-ma-001] and Destination MA [xxx-ma-001].

Userlevel 5
Badge +13

@AXVTKS 

Those errors “Failed to mount the disk media” are two part errors. There should be more to the error that states specifically what is wrong and why we failed to mount the storage resource. Can you post the full message you are receiving? Without this I wouldn’t be able to point you in a direction.

@Raj Balaraj 

Without logging we wouldn’t be able to say what the issue is. Can you see any errors in CvJobReplicatorODS.log on the source MediaAgent or CVD.log on the destination MediaAgent?

 

Thank you,
Collin

Userlevel 2
Badge +7
Error: Unable to send the stop data transfer control message to the tail. .

 

Userlevel 2
Badge +7
Failed to seek to the correct location on the media to start the data restore.

 

Userlevel 2
Badge +7

 

Userlevel 7
Badge +23

Sharing case resolution:

Solution:

Customer found after updating from 11.26.13 to 11.28.8 they were seeing numerous communications issues across multiple MAs. Issue was reviewed on session and escalated to Development. Development identified Additional Setting "sPipelineMode" was causing issue and recommended removing/disabling this. Once this was done Jobs started running without issue. Development continued to review for RCA but confirmed unable to determine without elevated logging and WireShark captures from time of issue. Discussed further with customer and confirmed case may be closed at this time as Jobs completing since removal of this additional setting.

Userlevel 4
Badge +12

I had a very similar issue(s) on 11:28

Case 1

Source (MagLib) and Destination Media (Tape)  agents were the same single machine

The incident occurred when a single Tape was being used (for example) STREAMS=1 (forcing all traffic to a single tape)

No Traffic would flow at all

Setting Streams to = 2  (closing the properties Window) - then opening it up again, and setting Streams = 1; and the AUX to Tape would run without issue

 

CASE 2:

13:138 error

Source (MagLib) and Destination Media (Tape)  agents were the same single machine - 

Again - writing to a single drive (forced via Streams = 1) 

Jobs would run - but would event report 

Error occurred while processing chunk [92nnnnnn] in media [V_nnnnnn], at the time of error in library [QQQQ] and mount path [[zzzzz], for storage policy [------] copy [--------------- etc . Unable to setup the copy pipeline. Please check connectivity between Source MA [1] and Destination MA [1].

Again - Toggling the Streams form 1 to 2 then back to 1  fixed the issue

This can be done while an AUX to tape is running, by suspending the job, ‘Toggling’ - resume the job, and the ‘pipeline error’ events are gone

 

Badge +1

Hello,

we are having the exact same problem between two mediaagents (error 13:138).

I am trying to locate the suggested solution consisting in disabling spipelinemode but I can’t find it. Where do you configure this setting?

This is the event thats repeats multiple times until the auxiliary copy fails:

Error occurred while processing chunk [1180897] in media [V_204964], at the time of error in library [Eseries2800] and mount path [[win-backup] C:\Librerias\Eseries\Vol01], for storage policy [Virtualizacion] copy [Copia_XX] MediaAgent [mediaagentaux]: Backup Job [399719]. Unable to send the stop data transfer control message to the tail.

 

Thanks

Badge

Hi All,

I am also having the simillar error with the auxilary copy and not able to find sPipelinemode.

please suggest where to find or look for this setting.

 

Thanks,

Anil

Userlevel 1
Badge +7

Hi Team,

I am also facing this similar error with while running the auxillary copy,(disk to Tape copy running)

Also please share me how to remove  "sPipelinemode” settings where this option located from commvault

please share me solution or suggestion for this issue, Please find the below error we are getting in commvault

Issue Description: 

Error Code: [13:138]
Description: Error occurred while processing chunk [2392546] in media [V_254576], at the time of error in library [NAS-DISKLIB01] and mount path [[MA01] \\IP\share01\NAS_Drive07], for storage policy [CommServeDR] copy [TapeCopy-ITS-CommServeDR] MediaAgent [MA01]: Backup Job [288446]. Unable to setup the copy pipeline. Please check connectivity between Source MA [MA01] and Destination MA [MA01].
Source: MA01, Process: CVJobReplicatorODS

Error Code: [62:114]
Description: There is no active controller for this library.
Source: csfloat, Process: MediaManager

Reply