Skip to main content
commvault.com commvault.com

This has been driving me mildly nuts for the past 3 days post a power shutdown for our site.

We had some challenges with our storage network, that prevented iSCSI comms for a few hours after we powered up our site post some scheduled electrical work.

We solved that by the software equivalent of a power on/power off on the switches in question (two unrelated port channels to our SAN were disabled/re-enabled). Great. iSCSI works again with all hosts.

The MA in question, a VM, is borrowing space on a temporary basis from a newer array, as I am doing a lot of migration work, and is mounting the volumes in Windows via iSCSI as opposed to using an iSCSI mounted data store in VMWare which is treated as an HDD by the OS.   

Keep in mind, this was all working swimmingly before this past weekend.  For the past three days, CV is convinced that the 5 mount points in question are offline, and do not have a controller.  If you go to the OS, and browse, you can drill down as deeply into the mounted volume as far as you want and see all that’s there.  

I’ve restarted CV.

I’ve rebooted the MA’s.

CV reports error 32:297, but, when you dig deeper, it’s really error 62:1458 -  The mount path is not present on MediaAgent”, but that’s an observable lie. 

I’ve tried running disk library maintenance on the library. No dice.  Here’s a view of what I see with some redacting.

 

 Any thoughts out there as to what the issue is?  What to try next?  This is breaking a lot of backups that are depending on the data in those 5 mount points for building synthetic images, and I can’t cleanly aux copy this stuff off to long term storage as well.

This part made my day “but that’s an observable lie. “ :rofl:

My initial suspicion is that they are presented to the MA on a different address than before, so CV is looking at them in a place they are not living, however, I’m going to rope in some experts here.

What do you see in the log files?  CVMA.log on the MA and MediaManager.log on the CS are a good place for us to start.

Mike,  

You’ll find I have rather wry sense of humour….it’s what kept me sane doing NetBackup support for more than 20 years. (I’m fairly agnostic about products, it was the 24/7 support that I needed to remain sane for)

That’s possible, but based on what I do know of our storage area networking, it’s a vlan, all static, no dhcp or dns, and doesn’t route to our main network. (I’m not the network guy, just the guy who abuses it. ;-)

I haven’t mastered using GxTail as yet (I’m a bit old school - I love using tools like TextPad for logs, so I may be missing some stuff, but here’s a fragment or what I can see:

 

3316  db8   10/28 10:58:01 -------  WORKER    ]  Failed to calculate the space usage for path cC:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28] in function rProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER    ]  Failed to calculate the space usage for path tC:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11] in function \ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   d  ]  Failed to calculate the space usage for path aC:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12] in function sProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   3  ]  Failed to calculate the space usage for path C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14] in function 4ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:01 -------  WORKER   b  ]  Failed to calculate the space usage for path C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43] in function \ProcessMagneticConfigRequest] exists or not in func, error=0x80070003:{CQiDirectory::GetSpaceUsage(494)/W32.3.(The system cannot find the path specified. (ERROR_PATH_NOT_FOUND.3))}
3316  db8   10/28 10:58:02 #######  WORKER   .263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28], ID h87], errCode U2], errorString tFailed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   n263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path eC:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11], ID o128], errCode \2], errorString AFailed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   n263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12], ID 129], errCode r2], errorString MFailed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   o263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14], ID M130], errCode r2], errorString [Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
3316  db8   10/28 10:58:02 #######  WORKER   M263397/0/0          ]  Failed to open Magnetic Config File for writing Mount Path information for Path 3C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43], ID r166], errCode 2], errorString Failed to open file C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43\CV_MAGNETIC\MountPathConfigs]. Continuing anyway
 

For the record, I’ve altered some naming conventions in that fragment… :-)

I love a good ‘sanity maintaining’ sense of humor!  Also, wise on changing the names to protect the innocent.  I advise the same thing for everyone here!

I found a kb for you here:

https://kb.commvault.com/article/MM0032

Need to recreate the mount path labels and you should be solid.

Keep me posted!

Thanks Mike,  I’ll give it a shot shortly. Clearly in this case your search fu is better than mind as this wasn’t turning up for me.

 

The good news is we are working on a whole new search experience across our customer portals that will make finding things across areas MUCH easier!

We will definitely be announcing it when it is ready!

Welp, that went splat.

Built the mount path text file as shown:

C:\Program Files\Commvault\ContentStore\Base>type c:\temp\newmountpathlabel.txt
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28\CV_MAGNETIC,1,87
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC,1,128
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12\CV_MAGNETIC,1,129
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14\CV_MAGNETIC,1,130
1AB0TF0,C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43\CV_MAGNETIC,1,166

 

Ran the command, results as shown:

C:\Program Files\Commvault\ContentStore\Base>CreateMountPathLabel.exe -if c:\temp\newmountpathlabel.txt

Processed line no: 1]Invalid path for creating Mount Path label file
>C:\ISCSI-MOUNTS\ARRAY4\Backstor04\Folder_07.08.2020_16.28\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: 2]Invalid path for creating Mount Path label file
>C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: 3]Invalid path for creating Mount Path label file
>C:\ISCSI-MOUNTS\ARRAY4\Backstor02\EOMNTJ_05.23.2021_20.12\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: 4]Invalid path for creating Mount Path label file
>C:\ISCSI-MOUNTS\ARRAY4\Backstor03\LM2W78_05.24.2021_17.14\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL

Processed line no: 5]Invalid path for creating Mount Path label file
>C:\ISCSI-MOUNTS\ARRAY4\Backstor05\G6VPPB_09.03.2021_21.43\CV_MAGNETIC] is invalid path for creating MOUNTPATHLABEL
 

I can confirm there is a MountPathConfigs file in the paths that won’t go online, there’s also a DrVolumeList and MOUNTPATH_LABEL file.  Those parts of the contents line up with the paths that are working, with the obvious delta’s reflecting the path id and actual path and are only found in the MountPathsConfigs files.

Working path file contents are:

<?xml version="1.0" encoding="UTF-8"?>
<MountPathSet>
<MountPath MountPathId='17'>
<MediaAgent HostId='4' AccessPath='C:\Mounts\ARRAY1_9\Folder_09.08.2017_20.58\CV_MAGNETIC' OrigCCId='2' OrigMPId='17'>
MEDIAAGENT01.mycompany.com
</MediaAgent>
</MountPath>
</MountPathSet>

 

Nonworking path file contents are:

<?xml version="1.0" encoding="UTF-8"?>
<MountPathSet>
<MountPath MountPathId='128'>
<MediaAgent HostId='4' AccessPath='C:\ISCSI-MOUNTS\ARRAY4\Backstor01\PWLZK5_05.23.2021_20.11\CV_MAGNETIC' OrigCCId='2' OrigMPId='128'>
MEDIAAGENT01.mycompany.com
</MediaAgent>
</MountPath>
</MountPathSet>

 

I think I’m going to have to punt this to support - I need to fix this before end of day today, and I am running out of time as there’s another bit of older storage gear that might need an angle grinder with an 80 grit flap disk to fix still ahead of me. 

 

David

Yeah, you read my mind.  If this is still giving us conflicting/unexpected results, open a case up.

Let me know the case number so I can follow up accordingly!

Mike - Not my first rodeo….:cowboy:   Please ref, Incident 211028-616

 

David

 

 

 

Thanks!  Reminds me of a joke I hear about how rodeos have a pretty low threshold for expertise….first rodeo, you’re forgiven.  but if it’s your 2nd rodeo, you should apparently know better!

:rofl:

:laughing:

Yah… Well, what can I say, I like to find new and novel ways to learn what better is - scientific method - quantify and qualify all the wrong ways and all the right ways.... 

Have a chat with Steve Jacobs here in your Canadian operation… he knows me… well. :-)

So. Solved. ISCSI mount point path name typo, in the middle of the now very long path string was causing the issue. repaired, restarted, all good again. (reason for the long path string before anyone asks, is we are trying to make it easier to correlate any volume mount on a system with it’s VMWare Datastore and it’s underlying disk array - and yes, that adds complexity, but that’s the price of a growing IT infrastructure).

Wow, good catch (and one that I never would have noticed)!

I’ll go ahead and mark your reply as the Best Answer to boot!

Reply


Terms & ConditionsCookie settings