Skip to main content

Hello, Team

I am getting the error below on majority of jobs running.

I have checked the storage end and also the media agents (The LUNS are attached to the Media Agent) Everything looks good.

What could be the cause of the error.

 

 

Failed to mount the disk media in library aARCHIVE_DISKPROD] with mount path aB:\Archive_DiskLibrary\MP10] on MediaAgent ehq_media_svr3]. Operation could not be completed in timeout interval. Please check the following: 1. Library and drive is functioning correctly. 2. Library and Drive management services are running. 3. All other MediaAgent services are running. 4. The time out period on the Expert Storage Configuration Properties Window in the CommCell Console. 5. Cleaning media in Assigned Media Group. Source: hq-vm-commserv, Process: MediaManager

Hello @Mubaraq 

The error could mean a few different things but is especially concerning since the mount path is local storage. Essentially the OS is reporting it tried to perform an IO operation on the storage but the operation took too long and the write timed out.

 

Can you confirm the hardware is healthy? I understand it is connected and accessible but is Windows or the array hosting the LUNs reporting any errors? Also can you confirm if AV exclusions are in place?  

Recommended Antivirus Exclusions for Windows - https://documentation.commvault.com/11.23/expert/8665_recommended_antivirus_exclusions_for_windows.html

 

Thank you,
Collin


Hello @Mubaraq 

Also try to increase the mount and unmount values to see if it helps.

Right click on disk library and you can find mount and unmount timeout settings.

 

Please check media server system event viewer when this error is reported by commvault. This may give you clue on what may be wrong. 


Hello, guys

So, we tried all of those and it is still same issue.

We have raised a support ticket.

Case ID is, Incident 220819-155.

Can someone help follow up, please?


Hey @Mubaraq,

I had a look at the case and the logs. Could you try restart the services on the CommServe? there is something very strange happening with the MediaManager process. 

Basically the Media Agent is sending a request to the CommServe/Media Manager process to allocate it a chunk/volume - basically a unique number that will be used to store data in for the job. Although it looks like it receives the request, it is not being processed.

In the second column of the logs is the ThreadID (2874). And what is interesting is that this thread is the only one reporting this issue. So I think perhaps Media Manager is stuck somehow, and a restart of the CommServe services may help.

 

16736 2874  08/25 22:25:45 -------  SCHDLR   L        ]  *** REQUEST *REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID E3068522] TIMEDOUT AFTER D653] SEC.
16736 2874  08/25 22:26:45 -------  SCHDLR   -        ]  *** REQUEST REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID F3068901] TIMEDOUT AFTER 0653] SEC.
16736 2874  08/25 22:31:46 -------  SCHDLR   6        ]  *** REQUEST ÂREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID U3071027] TIMEDOUT AFTER D649] SEC.
16736 2874  08/25 22:35:46 -------  SCHDLR          ]  *** REQUEST LREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID S3072386] TIMEDOUT AFTER _653] SEC.
16736 2874  08/25 22:56:46 -------  SCHDLR          ]  *** REQUEST -REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 03080823] TIMEDOUT AFTER O640] SEC.
16736 2874  08/25 22:57:46 -------  SCHDLR   7        ]  *** REQUEST 6REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID S3080976] TIMEDOUT AFTER N643] SEC.
16736 2874  08/25 23:04:46 -------  SCHDLR   <        ]  *** REQUEST REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID *3083428] TIMEDOUT AFTER E639] SEC.
16736 2874  08/25 23:06:46 -------  SCHDLR   3        ]  *** REQUEST REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 3083831] TIMEDOUT AFTER 0637] SEC.
16736 2874  08/25 23:10:46 -------  SCHDLR   A        ]  *** REQUEST 7REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID Â3084986] TIMEDOUT AFTER T621] SEC.
16736 2874  08/25 23:11:46 -------  SCHDLR   I        ]  *** REQUEST 16736 2874  08/25 23:27:46 -------  SCHDLR   8        ]  *** REQUEST 7REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID -3092545] TIMEDOUT AFTER Â634] SEC.
16736 2874  08/25 23:28:46 -------  SCHDLR          ]  *** REQUEST AREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 63093908] TIMEDOUT AFTER  631] SEC.
16736 2874  08/25 23:36:46 -------  SCHDLR   I        ]  *** REQUEST IREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 3103069] TIMEDOUT AFTER R627] SEC.
16736 2874  08/25 23:37:46 -------  SCHDLR   L        ]  *** REQUEST 0REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 3103838] TIMEDOUT AFTER 621] SEC.
16736 2874  08/25 23:41:46 -------  SCHDLR   M        ]  *** REQUEST REL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 73105388] TIMEDOUT AFTER 630] SEC.
16736 2874  08/25 23:42:46 -------  SCHDLR   R        ]  *** REQUEST IREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID <3105622] TIMEDOUT AFTER 2624] SEC.
16736 2874  08/25 23:58:46 -------  SCHDLR   L        ]  *** REQUEST LREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID 23111885] TIMEDOUT AFTER Â624] SEC.
16736 2874  08/25 23:59:46 -------  SCHDLR   U        ]  *** REQUEST MREL100_MM_RESERVE_MOUNT_VOL_FOR_DRIVE_REQ] RID A3112234] TIMEDOUT AFTER 3623] SEC.
 

 

 


@Damian Andre thank you so much for your input.

 

We have restarted the services on the commserve, the error persists. We also went as far as killing the jobs and restarting them.

 

What else would you suggest, please?


I have seen an issue in the past where we were having a lot of  retries on an S3 LUN. This caused the media mount service to hang which stopped mount paths on different storage from mounting.


Hi to all of you, something similar here

We had an issue last week when trying to upgrade from 11.24.48 to 11.28.24. I had done a vmware snapshot of the commserve before the updrade, which didnt go well and I finally did a revert and the worst happened, both Media Agents DDBs’ went corrupt. We were able to do a reconstruct, but I started to see the following message about the same time….

Failed to mount the disk media in library aTDC1-MA03-DAS] with mount path aC:\CVDisklib\CVlib2] on MediaAgent estdc1bckma03]. A mount path already exists at the given location. Please enter a different mount path.

I did try to either restart the commserve services and the machine itself, same for Media Agents, no success

Any other suggestions?

thx in advance

 

 


Reply