Data in some directories are not displayed (VSA Backup)

Thomas,

which Hypervisor are you using here? also need to know which hotfix pack you are on and where does the data sit on the backend. is this a cloud library or local disk library on prem?

Hi @Gseibak, 

we are using VMware ESXi, 6.5.0 and the Commvault Hotfix 11.20.40. The Data are stored on an DX100 Disk Storage on the backend. The library is ib prem and connected with the Media Agents via FC.

Thanks Thomas,

are you seeing any browse related errors in the FBR logs?

@Gseibak all I can see in the message log and the fbr.log is this and the time is wrong on the system but I think this could not be the problem.

Mar 10 10:07:25 cvfbr01 kernel: 369412: cvblk_create_block_device(): Creating store-backed RW recall block device FBR_1480585285-50052301-5955-71ad-2fb6-b88d485750aa-1
Mar 10 10:07:25 cvfbr01 kernel: 369412: cvblk_create_block_device(): Creating store-backed RW recall block device FBR_1480585285-50052301-5955-71ad-2fb6-b88d485750aa-2
Mar 10 10:07:25 cvfbr01 kernel: 369412: cvblk_create_block_device(): Creating store-backed RW recall block device FBR_1480585285-50052301-5955-71ad-2fb6-b88d485750aa-3
Mar 10 10:07:27 cvfbr01 kernel: cvblk1200: p1 p2 p3
Mar 10 10:07:34 cvfbr01 kernel: EXT4-fs (cvblk1200p1): mounted filesystem with ordered data mode. Opts: (null)
Mar 10 10:07:37 cvfbr01 kernel: EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null)
Mar 10 10:07:40 cvfbr01 kernel: EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null)
Mar 10 10:07:41 cvfbr01 kernel: EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: (null)
Mar 10 10:07:44 cvfbr01 kernel: EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null)
Mar 10 10:07:46 cvfbr01 kernel: EXT4-fs (dm-6): 5 orphan inodes deleted
Mar 10 10:07:46 cvfbr01 kernel: EXT4-fs (dm-6): recovery complete
Mar 10 10:07:46 cvfbr01 kernel: EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: (null)
Mar 10 10:07:48 cvfbr01 kernel: EXT4-fs (dm-7): 2 orphan inodes deleted
Mar 10 10:07:48 cvfbr01 kernel: EXT4-fs (dm-7): recovery complete
Mar 10 10:07:48 cvfbr01 kernel: EXT4-fs (dm-7): mounted filesystem with ordered data mode. Opts: (null)

The fbr log :
59149 5da6f 03/10 10:16:52 ####### CVWaitableTimer::UnixTimeoutHandler() - Inactivity Timeout canceled. So just bailing out
359149 5832e 03/10 10:16:52 ####### FBROnDemand::IncomingMsgXMLCallback() -  msg TagID received is [68dac6f7]
359149 5832e 03/10 10:16:52 ####### FBROnDemand::IncomingMsgXMLCallback() -  msg received is [<?xml version="1.0" encoding="UTF-8" standalone="no" ?><FBR_FBRFileSystemGetItemCountReq strPath="/root" uReqId="1480585085"/>]
359149 5832e 03/10 10:16:52 ####### FBROnDemand::IncomingMsgXMLCallback() - Handle FBRFileSystemGetItemCountReq
359149 5832e 03/10 10:16:52 ####### FBRServer::handleItemCountReq() - Request ID ==> 1480585085 Path ==> [/root]
359149 5832e 03/10 10:16:52 ####### FBRServer::handleItemCountReq() - Update lcl_cvblkMountPath : [/opt/FBR/cvblk_mounts/1480585085]
359149 5832e 03/10 10:16:52 ####### FBRServer::acquire_lock() - Acquiring lock for fbr_map
359149 5832e 03/10 10:16:52 ####### FBRServer::acquire_lock() - Got the lock for fbr_map
359149 5832e 03/10 10:16:52 ####### FBRServer::release_lock() - Released the lock for fbr_map
359149 5832e 03/10 10:16:52 ####### FBRServer::validate_enum_path() - ctx for [1480585085] is in acceptable state FBR_STATE_ATTACHED
359149 5832e 03/10 10:16:52 ####### FBRServer::validate_enum_path() - Path [/root] exists within mount [/opt/FBR/cvblk_mounts/1480585085/vg_system-root]
359149 5832e 03/10 10:16:52 ####### FBRServer::handleItemCountReq() - Updated path => [/opt/FBR/cvblk_mounts/1480585085/vg_system-root/root]
359149 5832e 03/10 10:16:52 ####### FBRServer::fill_items() - Update lcl_cvblkMountPath : [/opt/FBR/cvblk_mounts/1480585085]
359149 5832e 03/10 10:16:52 ####### FBRServer::handleItemCountReq() - set count [17] in count response
359149 5832e 03/10 10:16:52 ####### FBRServer::handleItemCountReq() - Response XML<?xml version="1.0" encoding="UTF-8" standalone="no" ?><FBR_FBRFileSystemGetItemCountResp ullCount="17"><req strPath="/root" uReqId="1480585085"/></FBR_FBRFileSystemGetItemCountResp>
359149 5832e 03/10 10:16:52 ####### FBRServer::handleItemCountReq() - Counted : [17] items for path [/opt/FBR/cvblk_mounts/1480585085/vg_system-root]
359149 5832e 03/10 10:16:52 ####### FBROnDemand::IncomingMsgXMLCallback() - Count done
359149 57b02 03/10 10:16:52 ####### ::ChkJobCompleteEvent() - Received Jobs Complete Event from Library. Starting Inactivity timer for On Demand Service [CVODS_fbr_cvfbr01_1]
359149 5a304 03/10 10:16:52 ####### FBROnDemand::IncomingMsgXMLCallback() -  msg TagID received is [9000103]
359149 5a304 03/10 10:16:52 ####### FBROnDemand::IncomingMsgXMLCallback() -  msg received is [<?xml version="1.0" encoding="UTF-8" standalone="no" ?><CNSession_PAbortRequest time="0"><status close="2" errnum="0" errtype="2"/></CNSession_PAbortRequest>]
359149 5da71 03/10 10:16:52 ####### CVWaitableTimer::UnixTimeoutHandler() - Inactivity Timeout canceled. So just bailing out
359149 57b02 03/10 10:16:52 ####### ::ChkJobCompleteEvent() - Received Jobs Complete Event from Library. Starting Inactivity timer for On Demand Service [CVODS_fbr_cvfbr01_1]
 

Thanks Thomas, I would also check Browse.log and if you dont see anything there we should get a ticket opened on it. Feel free to attach the log.

Hi Thomas,

this may be a “stupid” question - but why would you expect to see anything in /dev? /dev only contains device files which you can’t really restore anyway. So I wouldn’t be surprised to see this just being filtered in the view. Do you have other folders where you experience the same? (By the way /proc is also special - as is /sys) …

Best regards,

Christian

@Gseibak 
Where can I find the browse.log ? The Subclient contains over 100 VMs. Do I have view log on VM where I saw the problem ?

@Christian Kubik 
I thought i would be able to see all tthe files on the system wenn i browse through them. 
There is also a mount point I also can’t see the data. The vmdk of this mount point is backuped. And thats the real problem. 

Hi @thomas.S ,

that mount point definitely is an issue. What type of filesystem is configured on that mount point. What is the Linux version of the backed up VM vs. the linux version of the file recovery enabler?

I have seen XFS having issues if backed up VM is RHEL (or CentOS) 8 and the FREL is RHEL/CentOS 7 due to a change in the FS version where the older linux just can’t mount that newer FS.

@Christian Kubik 

The version of the fbr is:
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"

and the mount point should be ext4. I wait for the reply of the costomer to confirm this.

Hi Thomas, it should be on the server that is doing the browse. Are you using a FREL for this live browse? -https://docs.commvault.com/commvault/v11/article?p=31977.htm

FREL = File Recovery Enabler for Linux

@thomas.S for the missing mountpoint definitely make sure you are using a Centos 8 FREL if you are browsing XFS created on Centos 8.x guests.

The other point to check is if the mountpoint you used is listed under /cvlostandfound in the browse.

The FREL looks @ disk signatures, LVM ids and compares them against the fstab discovered on the guest. If there is a match we attempt to mount it under the original location. If not then all such mountpoints go under /cvlostandfound.

@Gseibak yes we are using FREL. The only place, if some errors occur during the live browse, can be stored on the FREL, right ?

and I just got feedback about the filesystem of the mount point. The answer was that ext4 was used for it. 

@thomas.S any errors would be logged to FBR.log on the FREL. 
 

also did you check under /cvlostandfound during the browse?

Hi @amitkar, 

I checked the FBR.log again but couldn't find any errors. I could not find the directory /cvlostandfound on the FBR.
Currently I am a bit at a loss here.

Regards 

Thomas

@thomas.S, the /cvlostandfound would be visible during the browse operation on the GUI ( not on the FREL itself ). 

Do you have an escalation in progress for this? Please let me know the case number if you have one and we’ll take a look @ the logs.

@thomas.S , let me know once you have a case created.  I’ll follow up and track the escalation so we can record the solution for posterity 

@thomas.S , did you get an incident created for this?  I looked for your cases and am not seeing one.

Hi @Mike Struening, 

no, we have not opened a case here yet.
Since I cannot open a case directly at Commvault, I will give the case to our service provider.

Hi @christophe, 

Danke für die Information. Dies erklärt das es hier wohl wirklich ein Problem vom FBR in der SP20 Version ist.

Regards 

Thomas

Appreciate the follow up, @thomas.S .  Once you have a case number, let me know and I’ll follow it.

@thomas.S following up to see if your Service Provider was able to open an incident (and if so, the result).

Thanks!

Dear all,

The patch was to fix an issue reading the devices in the /etc/fstab.

In their configuration, there was a device inside another device and we were mounting the second before mounting the first one. Therefore we couldn’t mount the second until the first one was mounted.

See ex below:

sdb

└─sdb1                   LVM2_member       0J3uTi-T0jC-hK1M-5KCi-nq26-lqX1-CCE6qc

  ├─mediavg-USB        xfs      e9f8d7s6d90g9j2k4   /home/Media/USB

  ├─mediavg-VIDEOS   xfs     65110cd9-90d6-440c-8ef1-189a06278ff3   /home/Media/USB/VIDEOS

In above ex, we were trying to mount mediavg-VIDEOS before mediavg-USB and of course it failed.

So if the rules are followed and have a similar configuration in the /etc/fstab then yes, please log a case and we will check and confirm the patch is needed.

Best Regards,

Seb.

I meant to add, the fix is already official and public in SP20-HotFix-2442 in 20.19.

So just update the FREL to MR19 or above and confirm if the issue is fixed.

If /etc/fstab is configured properly and issue is not fixed, then please log a case and we will check.

Best Regards,

Seb