MediaAgent iSCSI disk library mount marked offline, but is present.

This part made my day “but that’s an observable lie. “ 

My initial suspicion is that they are presented to the MA on a different address than before, so CV is looking at them in a place they are not living, however, I’m going to rope in some experts here.

What do you see in the log files?  CVMA.log on the MA and MediaManager.log on the CS are a good place for us to start.

Mike,  

You’ll find I have rather wry sense of humour….it’s what kept me sane doing NetBackup support for more than 20 years. (I’m fairly agnostic about products, it was the 24/7 support that I needed to remain sane for)

That’s possible, but based on what I do know of our storage area networking, it’s a vlan, all static, no dhcp or dns, and doesn’t route to our main network. (I’m not the network guy, just the guy who abuses it. ;-)

I haven’t mastered using GxTail as yet (I’m a bit old school - I love using tools like TextPad for logs, so I may be missing some stuff, but here’s a fragment or what I can see:

3316  db8   10/28 10:58:01 -------  WORKER   [  ]  Failed to calculate the space usage for path [C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28] in function [ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   [  ]  Failed to calculate the space usage for path [C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11] in function [ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   [  ]  Failed to calculate the space usage for path [C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12] in function [ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   [  ]  Failed to calculate the space usage for path [C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14] in function [ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   [  ]  Failed to calculate the space usage for path [C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43] in function [ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:02 #######  WORKER   [263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path [C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28], ID [87], errCode [2], errorString [Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   [263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path [C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11], ID [128], errCode [2], errorString [Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   [263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path [C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12], ID [129], errCode [2], errorString [Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   [263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path [C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14], ID [130], errCode [2], errorString [Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   [263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path [C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43], ID [166], errCode [2], errorString [Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
 

For the record, I’ve altered some naming conventions in that fragment… :-)

I love a good ‘sanity maintaining’ sense of humor!  Also, wise on changing the names to protect the innocent.  I advise the same thing for everyone here!

I found a kb for you here:

https://kb.commvault.com/article/MM0032

Need to recreate the mount path labels and you should be solid.

Keep me posted!

Thanks Mike,  I’ll give it a shot shortly. Clearly in this case your search fu is better than mind as this wasn’t turning up for me.

The good news is we are working on a whole new search experience across our customer portals that will make finding things across areas MUCH easier!

We will definitely be announcing it when it is ready!

Welp, that went splat.

Built the mount path text file as shown:

C:\Program Files\Commvault\ContentStore\Base>type c:\temp\newmountpathlabel.txt
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28\CV_MAGNETIC,1,87
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC,1,128
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12\CV_MAGNETIC,1,129
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14\CV_MAGNETIC,1,130
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43\CV_MAGNETIC,1,166

Ran the command, results as shown:

C:\Program Files\Commvault\ContentStore\Base>CreateMountPathLabel.exe -if c:\temp\newmountpathlabel.txt

Processed line no: [1]Invalid path for creating Mount Path label file
[C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: [2]Invalid path for creating Mount Path label file
[C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: [3]Invalid path for creating Mount Path label file
[C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: [4]Invalid path for creating Mount Path label file
[C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: [5]Invalid path for creating Mount Path label file
[C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL
 

I can confirm there is a MountPathConfigs file in the paths that won’t go online, there’s also a DrVolumeList and MOUNTPATH_LABEL file.  Those parts of the contents line up with the paths that are working, with the obvious delta’s reflecting the path id and actual path and are only found in the MountPathsConfigs files.

Working path file contents are:

<?xml version="1.0" encoding="UTF-8"?>
<MountPathSet>
<MountPath MountPathId='17'>
<MediaAgent HostId='4' AccessPath='C:\Mounts\ARRAY1_9\Folder_09.08.2017_20.58\CV_MAGNETIC' OrigCCId='2' OrigMPId='17'>
MEDIAAGENT01.mycompany.com
</MediaAgent>
</MountPath>
</MountPathSet>

Nonworking path file contents are:

<?xml version="1.0" encoding="UTF-8"?>
<MountPathSet>
<MountPath MountPathId='128'>
<MediaAgent HostId='4' AccessPath='C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC' OrigCCId='2' OrigMPId='128'>
MEDIAAGENT01.mycompany.com
</MediaAgent>
</MountPath>
</MountPathSet>

I think I’m going to have to punt this to support - I need to fix this before end of day today, and I am running out of time as there’s another bit of older storage gear that might need an angle grinder with an 80 grit flap disk to fix still ahead of me. 

David

Yeah, you read my mind.  If this is still giving us conflicting/unexpected results, open a case up.

Let me know the case number so I can follow up accordingly!

Mike - Not my first rodeo….   Please ref, Incident 211028-616

David

Thanks!  Reminds me of a joke I hear about how rodeos have a pretty low threshold for expertise….first rodeo, you’re forgiven.  but if it’s your 2nd rodeo, you should apparently know better!

Yah… Well, what can I say, I like to find new and novel ways to learn what better is - scientific method - quantify and qualify all the wrong ways and all the right ways.... 

Have a chat with Steve Jacobs here in your Canadian operation… he knows me… well. :-)

Wow, good catch (and one that I never would have noticed)!

I’ll go ahead and mark your reply as the Best Answer to boot!