Description: Error occurred while processing chunk - error Code: [13:138]
Hi all!
could you advice me how to troubleshoot following types of error:
Error Code: 13:138] Description: Error occurred while processing chunk xxx] in media xxx], at the time of error in library disklib01] and mount path [xxx] /srv/commvault/disklib01/xxx], for storage policy XXX] copy Xxx] MediaAgent svma1]: Backup Job xxx]. Unable to setup the copy pipeline. Please check connectivity between Source MA svma1] and Destination MA svma1].
At a glance, it seems that it is not possible for CV to process chunk from the (index?)/disk library...However, the issue is connected with storage policy copy, that moves data from the disk library to the tape library (secondary copy). The main problem for us is that it is not possible to copy data to the tapes. Therefore, it may say Unable to setup the copy pipeline. The media agent is one server/device, that communicates with both disk and tape library. Lastly, the files in the related directories dont seem to be corrupted...
Any suggestions and ideas will be appreciated!
Page 1 / 1
After upgrading from 11.26.13 to 11.28.8 we began experiencing frequent errors on aux copy jobs, at times multiple errors per hour.
CV support escalated our support ticket to development. We had an existing additional setting on our HyperScale Media agents. The setting was sPipelineMode and it was set to AB:P
CV Development requested that we try removing this setting. After removing this setting the chunk errors with code 13:138 stopped occurring as did two other errors. The other two errors were timeout errors code 13:85 and unknown buffer type [0] received code 7:42.
Same issue happening now to me..it is for Tape library copy. Is the reason the drive need replace / faulty? is that a root cause (CRC error)? Error: Unable to setup the copy pipeline. Please check connectivity between Source MA [xxx-ma-001] and Destination MA [xxx-ma-001].
Now I have the same problem, the version between MA is the same but for hotfix is different. Will different hotfixes cause the same error?
I had a very similar issue(s) on 11:28
Case 1
Source (MagLib) and Destination Media (Tape) agents were the same single machine
The incident occurred when a single Tape was being used (for example) STREAMS=1 (forcing all traffic to a single tape)
No Traffic would flow at all
Setting Streams to = 2 (closing the properties Window) - then opening it up again, and setting Streams = 1; and the AUX to Tape would run without issue
CASE 2:
13:138 error
Source (MagLib) and Destination Media (Tape) agents were the same single machine -
Again - writing to a single drive (forced via Streams = 1)
Jobs would run - but would event report
Error occurred while processing chunk 92nnnnnn] in media V_nnnnnn], at the time of error in library iQQQQ] and mount path t zzzzz], for storage policy p------] copy ]--------------- etc . Unable to setup the copy pipeline. Please check connectivity between Source MA r1] and Destination MA i1].
Again - Toggling the Streams form 1 to 2 then back to 1 fixed the issue
This can be done while an AUX to tape is running, by suspending the job, ‘Toggling’ - resume the job, and the ‘pipeline error’ events are gone
@drPhil , based on the error, there’s a communication issue:
between Source MA Msvma1] and Destination MA Msvma1].
Can you check the connection reliability between those 2?
Hello @drPhil
You cannot re-associate a copy to a different Global Secondary Copy once its been associated.
You can put the issue drive into maintenance so that we won’t try to use it and you should be able to add the second scratch pool to the data paths of the Global Secondary Policy.
I concur with @RyanOJD . If you are in the same boat version wise, create a support case.
If you can both share the incident number, I can track them.
Incident ID: 220712-540.
@drPhil , based on the error, there’s a communication issue:
between Source MA Msvma1] and Destination MA Msvma1].
Can you check the connection reliability between those 2?
Hi @Mike Struening, I hope you are doing well
The thing is that there is only one Media Agent. Therefore, it looks a little bit incomprehensible.
After upgrading from 11.26.13 to 11.28.8 we began experiencing frequent errors on aux copy jobs, at times multiple errors per hour.
CV support escalated our support ticket to development. We had an existing additional setting on our HyperScale Media agents. The setting was sPipelineMode and it was set to AB:P
CV Development requested that we try removing this setting. After removing this setting the chunk errors with code 13:138 stopped occurring as did two other errors. The other two errors were timeout errors code 13:85 and unknown buffer type [0] received code 7:42.
We were asked to add this setting as well during troubleshooting for another ticket we had open. this caused all kinds of chunk errors and unknown buffer type errors during running of aux copies to our immutable storage. it was also causing the jobs to hang and not complete or fail. we removed this setting as well and now we are no longer getting these errors and our jobs are completing again.
Hi Team,
I am also facing this similar error with while running the auxillary copy,(disk to Tape copy running)
Also please share me how to remove "sPipelinemode” settings where this option located from commvault
please share me solution or suggestion for this issue, Please find the below error we are getting in commvault
Issue Description:
Error Code: w13:138] Description: Error occurred while processing chunk 2392546] in media V_254576], at the time of error in library iNAS-DISKLIB01] and mount path t MA01] \\IP\share01\NAS_Drive07], for storage policy pCommServeDR] copy ]TapeCopy-ITS-CommServeDR] MediaAgent aMA01]: Backup Job u288446]. Unable to setup the copy pipeline. Please check connectivity between Source MA rMA01] and Destination MA iMA01]. Source: MA01, Process: CVJobReplicatorODS
Error Code: 062:114] Description: There is no active controller for this library. Source: csfloat, Process: MediaManager
Hi All,
I am also having the simillar error with the auxilary copy and not able to find sPipelinemode.
please suggest where to find or look for this setting.
Thanks,
Anil
Hello,
we are having the exact same problem between two mediaagents (error 13:138).
I am trying to locate the suggested solution consisting in disabling spipelinemode but I can’t find it. Where do you configure this setting?
This is the event thats repeats multiple times until the auxiliary copy fails:
Error occurred while processing chunk k1180897] in media aV_204964], at the time of error in library yEseries2800] and mount path h win-backup] C:\Librerias\Eseries\Vol01], for storage policy yVirtualizacion] copy yCopia_XX] MediaAgent tmediaagentaux]: Backup Job b399719]. Unable to send the stop data transfer control message to the tail.
Thanks
What version of CV are you on? We started having these same errors after upgrading to 11.28.8.
We have the same error message intermittently occur where the media agent can’t communicate with itself.
Unable to setup the copy pipeline. Please check connectivity between Source MA Mma004] and Destination MA Mma004].
We do have an active ticket with support.
Sharing case resolution:
Solution:
Customer found after updating from 11.26.13 to 11.28.8 they were seeing numerous communications issues across multiple MAs. Issue was reviewed on session and escalated to Development. Development identified Additional Setting "sPipelineMode" was causing issue and recommended removing/disabling this. Once this was done Jobs started running without issue. Development continued to review for RCA but confirmed unable to determine without elevated logging and WireShark captures from time of issue. Discussed further with customer and confirmed case may be closed at this time as Jobs completing since removal of this additional setting.
I’m seeing the same error messages as well.
@AXVTKS
Those errors “Failed to mount the disk media” are two part errors. There should be more to the error that states specifically what is wrong and why we failed to mount the storage resource. Can you post the full message you are receiving? Without this I wouldn’t be able to point you in a direction.
@Raj Balaraj
Without logging we wouldn’t be able to say what the issue is. Can you see any errors in CvJobReplicatorODS.log on the source MediaAgent or CVD.log on the destination MediaAgent?
Thank you, Collin
@drPhil , I must be blind
Apologies for missing the obvious!
I concur with @RyanOJD . If you are in the same boat version wise, create a support case.
If you can both share the incident number, I can track them.
Hello @Collin Harper
We have seen this very error, “13:138 Error Occurred while processing chunkh...]”
/EDIT] Adding one more error message seen
“Failed to mount the disk media in library l…] with mount path o…] on mediaagent …]”
except this is during a job that backs up to an s3 bucket.
Unfortunately I dont have access to the CV application myself, this is what I know of the problem and trying to see how can it be helped.
I dont know if there are tape libraries in the picture at the moment.
Is there anything you can point me to?
Thank you in advance!
Hi all,
after investigating we have found out that there is an issue with the tape drive, that shows some errors on the management panel. I think, the error posted above (Error while processing chunk) is not related with the new version of Commvault since we have 11.24.48.
Our next step will be the most likely to obtain new tape drive. However, is it possible to change Global Secondary Policy Copy in the Policy Copy setting tab (it is grey out)? Now, we would need to use only one tape drive, but it is necessary to let the Policy Copy to choose different scratch pool...And Global Secondary Policy Copy can have only one scratch pool defined…..