Solved

Folder missing in Intellisnap Netapp Snapshot


Userlevel 1
Badge +4

Hi,

I have a strange behavior with Commvault and Intellisnap. We trigger Netapp Snapshots with Commvault Jobs. These Snaps are not copied to Commvault Library. Reason: We have a 2-Node Netapp Metrocluster in Place and a third Netapp with async Snapmirror syncronisation to keep the History of Snapshots there (so this is kind of a Backup Storage for the Metrocluster).

We use Commvault only for triggering Snaps and for restores from these snaps. But Data is kept on Netapp with Snapshots. So this way our Service Desk personell needs only one GUI to restore Backups from Netapp and from all other kind of Backups we store in Commvault Library.

Since about 3 Months I observe the following problem: When you browse a Netapp Snapshot Job with Commvault, you don’t see all Folders. When you browse the same Snapshot with Windows Explorer using ~snapshot after the UNC path, everything is there and you can restore it with no problem.

Any idea why the Folders are not visible in Commvault Job browsing? I do not index these backup jobs. The snaps will be mounted to Media Agent and live browsed when I choose “browse job”.

best regards
Patrick

icon

Best answer by TheDude75 13 September 2021, 15:01

View original

18 replies

Userlevel 6
Badge +14

Hi Patrick,

 

I suspect it could be the Page Size of the Browse. - When you perform a Browse, click the Advanced Options tab and enter a larger value for Page Size. i.e. 999999999

Let us know how you get on.

 

Best Regards,

Michael

Userlevel 1
Badge +4

Hi Michael,

Thanks for your reply. Unfortunately there are not that many folders in the path so Page size makes no difference. And folders are randomly missing. For example:

aaabbb → visible
bbccccc → missing
ddd → visible
eee → visible
...

I still did not found out why some folders are missing and others are not. The majority of folders is listed but some are not. I checked multiple jobs. Looks like the missing folders are not visible in any job.

I try to find the difference between visible and not visible folders now. I will keep you updated.

best regards
Patrick

 

Userlevel 1
Badge +4

totally weird. I found out:

visible Folders: The owner in NTFS is a User (it does not matter what user)

not visible Folders: The owner in NTFS is the Administrators Group of the Netapp Storage

I will check this with a colleague who did the Data Migration to Netapp and was responsible for setting the ownership of migrated data. Something must be wrong with this ownership.

Userlevel 7
Badge +23

Nice catch, @TheDude75 !  Please come back and share what you find out about the ownership details.

Userlevel 6
Badge +14

Thanks for the update,

 

Is the user (Used for Backups) in the Subclient a member of the NetApp Administrators Group?

 

Best Regards,

Michael

Userlevel 7
Badge +21

I don't think owner matters as much, what are the other ACLs on the folder/

When you are browsing the snapshot file system, it seems like the account context running under the media agent may not have access due to those permissions. I would actually think Commvault is looking at the folder structure under the system context. Do the folders allow for system / local system?

Userlevel 1
Badge +4

Is the user (Used for Backups) in the Subclient a member of the NetApp Administrators Group?

In array management, I have the Netapp local “vsadmin” User used for connecting the Netapp device.

this User should be administrator on the Netapp Storage.

 

Userlevel 1
Badge +4

Do the folders allow for system / local system?

Yes, SYSTEM has Full control on NTFS.

Userlevel 7
Badge +23

@TheDude75 , any further word from your colleague?  Anything discovered as the source?

Userlevel 1
Badge +4

I still try to find out what’s the difference of these visible and invisible folders. It was not the owner or permissions. I found out the creation date of the invisible folders is always the same (August 2020). The visible folders are newer. So i think during robocopy of these Files to the share in 2020, they did something strange. The newer folders are created manually and work correctly.

I still wait for my colleague to analyse this further (he is the guy who migrated the data to this share with a script in august 2020).

Userlevel 7
Badge +23

Wow that’s a good catch on the date.  You likely found the issue.  Looking forward to the cause if you can catch that as well!

Userlevel 7
Badge +23

Hey @TheDude75 , following up to see if you ever nailed down the cause of the difference between the folders.

Thanks!!

Userlevel 1
Badge +4

Hi everybody. Problem is still ongoing. But we found out more:

  • If I rename a folder on the filesystem and do a backup again. This folder is visible in the new backup job.
  • If I rename the folder back to its original name and do another backup. It’s NOT visible again when browsing the latest snapshot.

so it looks like it’s some kind of cache in Commvault or something. This is really weird. But I can reproduce it 100% with every invisible folder. It works back and forth when renaming the folders again. So I don’t think there is something wrong with the filesystem. It’s really a problem in Commvault and how it displays the content of the snapshot.

I opened a support case again and we will have a session today.

I keep you updated.

Userlevel 7
Badge +23

Appreciate the update!  Can you share the case number so I can follow up accordingly?

Userlevel 1
Badge +4

It’s # 210827-160

Userlevel 7
Badge +23

Thanks!!  I see you are working with James.  He’s an EXTREMELY sharp guy!!

Userlevel 1
Badge +4

Hi Everybody,

Problem is solved. Commvault Developers created a patch: v11SP20_Available_Diag4193_WinX64.exe

After the installation of this patch on my browsing Media Agents the problem was fixed.

 

Userlevel 7
Badge +23

Thanks for sharing, @TheDude75 !!

Reply