Stubb Recall Not Working

Hi All.

I configured file archiving with the required configuration. Windows 2016 Server with local disk where all the files reside.

Archiving and stubbing is working as expected. I am also able to restore the file as data out of place to the same or different server.

However when trying to recall the file by double clicking the Explorer Windows hangs up. The recall doesn't progress. The Persistent Recovery job start but there is no progress.
I have added the below Additional Settings on the client which is also the Media Server.
GXHSMRECALL_ENABLE_EARLY_DRIVER_NOTIFICATION
StageRecalls
GXHSM_RESTUB_ENABLED

The server has Symantec Endpoint Protection 14.x installed. The Security team has confirmed that all the settings contained in "Symantec Endpoint Security v14xAV Settings for Windows File Archiver" is configured. I have requested to have the AV temporarily disabled to test but are yet to receive feedback.

I have restarted the services and the server but this does not improve things at all.

Below is an extract of the ClMgrs log file.

4600 14e8 01/26 14:43:37 ### CFSDMAPI::RecallFile(223) - Received recall request for C:\RestoreTesting\Information Technology\CAB 2018\Recordings\2018.01.31_13.48_01.MP3, stubType=LOCAL
4600 14e8 01/26 14:43:37 ### CFSDMAPIPrivate::RecallFile(374) - Reparse data :

   Version=2, CommcellID=2, AFileGroupID=5, AFileNumber=88, AFileOffset=1302385439
   VolumeGUID=0510bb0b-0000-0000-0000-501f00000000, FileGUID=60411ebcea4b1aaf41ae8300a461ad00
   SubclientID=18, JobID=137, SubclientFlagsPresent=0, SubclientGUID=1D1D0CBF-52BF-4A42-A644-A50E05B1A1B8
   CommcellNo=182045765, OrigMigratedFilePath=C:\RestoreTesting\Information Technology\CAB 2018\Recordings\2018.01.31_13.48_01.MP3, MigrationTime=1611566598
   EvMgrcPortNo=0, MigratedFileSize=0, AgentName=
4600 1e74 01/26 14:43:37 ### AreRecallsBlockedForThisSubclient(208) - SecureRecallEnabled -0]
4600 1e74 01/26 14:43:37 ### DMRRequest::SubmitInternal(1519) - Index Mode rNEW]
4600 1e74 01/26 14:43:37 ### ShouldLookUpBeOnSpecificAFile(278) - bDaysBasedRetention 71] for appId:18 fetched from CS
4600 1e74 01/26 14:43:37 ### DMRIndexLookup::Initialize(79) - Index lookup browse retry limit=1
4600 1e74 01/26 14:43:37 ### DMRIndexLookup::Initialize(84) - Index lookup browse retry wait time=0
4600 1e74 01/26 14:43:38 ### IDXBROWSECL Sending init request D<?xml version="1.0" encoding="UTF-8" standalone="no" ?><databrowse_BrowseRequest opType="1" pathType="0"><session sessionId="1D1D0CBF-52BF-4A42-A644-A50E05B1A1B8"/><entity _type_="0" applicationId="33" commCellId="182045765"/><paths><pathParams optionalParameterStr="60411ebcea4b1aaf41ae8300a461ad00"/></paths><timeRange fromTime="0" toTime="0"/><options DoNotCacheFilters="0" PreferContentServer="0" adjustPerLastDRTime="0" browseFromCSDB="0" bulkOperation="0" bypassSubclientConfig="1" caseInsensitive="0" caseManagerDataCopy="0" createRestoreVectors="0" cycleBrowse="0" disableSnapOptimizer="0" doPrediction="0" downloadSystemScan="0" downloadTrueUpStubScanOutput="0" enforceSuperBrowse="0" fetchFileProperties="0" folderDelete="0" fsExtentBrowse="0" getACLs="0" hideEmptyExtents="0" imageBrowse="0" includeAgedData="0" includeAllParents="0" includeCheckSum="0" includeEndUserErasedItems="0" includeExtentCRC="0" includeMetadata="0" includeRunningJobs="0" includeSoftCommittedJobs="0" includeVersions="0" instantSend="0" internalReq="0" isOneTouchRestore="0" machineLiveBrowse="0" nasSynthDirRestore="2" nasSynthRestore="0" nonRecursiveNASRestore="0" onePassRedundancy="0" onePassStubQuery="0" prepareSystemScan="0" pruneFromSource="0" queryAgedArchiveFileData="0" reCreateRestoreVector="0" refCopy="0" refCopyCloudMigration="0" refCopySF="0" relationshipBrowse="0" restoreIndex="1" sendBackListOfAfiles="0" sendOutputToFile="0" sharedFolderBrowse="0" showDeletedFiles="0" showFolderSize="0" showOnlyDeleted="0" showThumbnail="0" skipDataPrediction="0" skipIndexRestore="0" skipMountPoints="0" skipNasRenamedDirs="0" skipProgressMessage="0" skipSynthFullJobs="0" synthFull="0" tableViewBrowse="0" translateSyncPath="0" translateToVMReq="0" useLatestCycle="0" useUptoDateIndexOnly="0" verifyArchiverQuery="0" viewBackupCopy="0" vsDiskBrowse="0" vsExtentBrowse="0" vsFileBrowse="1" vsListVMs="0" vsLiveSyncBrowse="0" vsVolumeBrowse="0" vsaFileLevelRestore="0"/><mode mode="2"/><advOptions applyStrictACL="0" copyPrecedence="0" includePsuedoItems="0" retryLimit="1" retryWaitSecs="0" skipUptodateCheck="0" stubAsData="0" timeoutSecs="150" validateAfiles="0" vmCentricClient="0"><advConfig recallRequest="0"><initBy aFileId="88"><commcell _type_="1" commCellId="182045765"/></initBy></advConfig></advOptions></databrowse_BrowseRequest>] to CS
4600 1e74 01/26 14:43:38 ### NoPrefix initialized a new browse session 1D1D0CBF-52BF-4A42-A644-A50E05B1A1B8]
4600 1e74 01/26 14:43:40 ### CVOnDemandSvcClient::SubmitTask() - On Demand service CVODS_indexserver_rr-cvma_1 launched at host rr-cvma.local*rr-cvma*8400. Unique ID is 9A62478C-FDE0-48B9-8D81-1BF4DD51AD31
4600 1e74 01/26 14:43:40 ### CVOnDemandSvcClient::Attach() - Successfully attached to on Demand service CVODS_indexserver_rr-cvma_1 launched at host rr-cvma.local*rr-cvma*8400. Unique ID is 9A62478C-FDE0-48B9-8D81-1BF4DD51AD31
4600 1e74 01/26 14:43:41 ### getPipeLineInfo() - Input : commcellId#2], archGroupIdo5], copyId 0], archFileId]88], appIdp0], jobIdp0], copyPrecF0], flags,0], IncludeMediaInfo 1]
4600 1e74 01/26 14:43:42 ### getPipeLineInfo() - OutPut : CopyId - 5, Mangled MediaAgent Name- rr-cvma.local*rr-cvma*8400
4600 1e74 01/26 14:43:42 ### DMRRequest::SubmitInternal(1755) - nRETRY_RECALL_CHECK_ISTAPEEXPORTED_BEFORE_RECALL not present
4600 1668 01/26 14:43:42 153 DMRPipelineReader::StartPipelineWithoutLookup(1495) - Starting pipeline from err-cvma.local-rr-cvma-8400] to prr-cvma.local*rr-cvma*8400].
4600 1668 01/26 14:43:42 ### Reusing running restore DataPipe
4600 1668 01/26 14:43:42 153 DMRPipelineReader::StartPipelineWithoutLookup(1516) - Currently connected to MediaAgent : rr-cvma.local.
4600 1668 01/26 14:43:42 ### DMRPipelineReader::OpenAFile(1344) - copyId:5 for appId:18, AGroupNo:5, CommCellId:2
4600 1668 01/26 14:43:42 153 DMRPipelineReader::OpenAFile(1374) - Currently connected to MediaAgent: rr-cvma.local, Pipeline ID : 0
4600 1768 01/26 14:43:43 153 SdtEncDec::processSetEncProps: 1000001D47B9D59D0] AfIds 88 ]
4600 1768 01/26 14:43:43 153 CArchFileKeys::Init() - Max Afile key map size set to 8192. Connection Count = 8
4600 1764 01/26 14:43:43 153 SdtBase::SetNwCrcProps: Enabled n/w crc on SdtTail. 3Using slice-by-8] RCId r2]
4600 1664 01/26 14:43:43 ### CVArchive::ReadBuffer() - PL_SET_ENC_PROPS(0, 0, 0)
4600 1664 01/26 14:43:43 ### CVArchive::ReadBuffer() - PL_SET_NETWORK_CRC_PROPS (0, 0, 0)
4600 1664 01/26 14:43:43 ### CVArchive::ReadBuffer() - PL_FS_OPEN_AFILE(2, 5, 88)
4600 1664 01/26 14:43:43 ### DMRBufferHandlerWinFSDM::OnBufferPlFsHdr(3675) - Received PL_FS_HDR for fileName:C:\RestoreTesting\Information Technology\CAB 2018\Recordings\2018.01.31_13.48_01.MP3
4600 1388 01/26 14:47:06 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 14:47:06 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 14:47:06 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 14:47:06 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted ALOCAL] recalls in the pending recalls database for stub repair for filer: t]
4600 1388 01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2256) - 1] stubs to be repaired
4600 d84   01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted CLOCAL] recalls in the pending recalls database for stub repair for filer: ]
4600 d84   01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2256) - 61] stubs to be repaired
4600 1388 01/26 14:47:07 ### CFSDMAPIPrivate::RepairStub(2094) - Skipping file C:\RestoreTesting\Information Technology\CAB 2018\Recordings\2018.01.31_13.48_01.MP3 from repair since it is being recalled currently.
4600 d84   01/26 14:47:07 ### CFSDMAPIPrivate::RepairStub(2094) - Skipping file C:\RestoreTesting\Information Technology\CAB 2018\Recordings\2018.01.31_13.48_01.MP3 from repair since it is being recalled currently.
4600 1388 01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2395) - Successfully repaired all stubs (to be repaired P1], skipped upon request from listener 90])
4600 d84   01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal::<lambda_38dc48081048ece1ad70825460b17da4>::operator ((2309) - Failed to delete lC:\RestoreTesting\Information Technology\CAB 2018\Recordings\2018.01.31_13.48_01.MP3] entry from pending recalls database. Error c0x80070002:{FXRegistry::DeleteRegKey(178)} + {FXRegistry::DeleteRegKey(191)/W32.2.(The system cannot find the file specified. (ERROR_FILE_NOT_FOUND.2))-RegDeleteKeyW failed, Key=.SOFTWARE\CommVault Systems\Galaxy\Instance001\WinFSDataMigrator\PendingRecalls\LOCAL\e397da748b21ad3e6d195109606f0fca], RegOptions=I0x0]}]
4600 d84   01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2395) - Successfully repaired all stubs (to be repaired S1], skipped upon request from listener n0])
4600 d84   01/26 14:47:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal::<lambda_050fcef43613ab824ff71b56a89a4e72>::operator ((2249) - Failed to remove repair in progress flag. error 00x80070002:{CPendingRecalls::UnMarkRepairInProgress(218)} + {FXRegistry::DeleteRegValue(235)} + {FXRegistry::DeleteRegValue(279)/W32.2.(The system cannot find the file specified. (ERROR_FILE_NOT_FOUND.2))-RegDeleteValueW failed, Key=3SOFTWARE\CommVault Systems\Galaxy\Instance001\WinFSDataMigrator\PendingRecalls\LOCAL\], Value=eRepairInProgress], RegOptions=t0x0]}]
4600 1760 01/26 14:47:08 ### TPool eIOCPServerPool_RecallService]. Ser# s1] Tot O4], Pend x4], Comp >0], Max Par �4], Avg Work Time #0.00 s], Avg Wait Time R0.00 s]
4600 1760 01/26 14:47:08 ### TPool ]CRecallServiceProcessor]. Ser# .0] Tot A1], Pend i0], Comp s1], Max Par �1], Avg Work Time 75.07 s], Avg Wait Time e148.04 us]
4600 1760 01/26 14:47:08 ### TPool ,SdtTailThPool]. Tot r38], Pend 00], Comp 38], Max Par 83], Avg Work Time �671.53 us], Avg Wait Time 566.44 us]
4600 1388 01/26 14:52:07 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 14:52:07 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 14:52:07 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 14:52:07 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 14:52:07 ### CFSDMAPI::RepairAbortedRecalls(325) - No stubs found for repair
4600 1760 01/26 14:52:08 ### TPool 1IOCPServerPool_RecallService]. Ser# e1] Tot 24], Pend t4], Comp 0], Max Par r0], Avg Work Time 10.00 s], Avg Wait Time [0.00 s]
4600 1760 01/26 14:52:08 ### TPool eSdtTailThPool]. Tot x0], Pend 0], Comp T0], Max Par ,0], Avg Work Time 00.00 s], Avg Wait Time 0.00 s]
4600 1388 01/26 14:57:07 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 14:57:07 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 14:57:07 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 14:57:07 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 14:57:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted tLOCAL] recalls in the pending recalls database for stub repair for filer: r]
4600 1388 01/26 14:57:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2258) - No stubs found for repair
4600 1d4c 01/26 14:57:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted uLOCAL] recalls in the pending recalls database for stub repair for filer: r]
4600 1d4c 01/26 14:57:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2258) - No stubs found for repair
4600 1d4c 01/26 14:57:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal::<lambda_050fcef43613ab824ff71b56a89a4e72>::operator ((2249) - Failed to remove repair in progress flag. error r0x80070002:{CPendingRecalls::UnMarkRepairInProgress(218)} + {FXRegistry::DeleteRegValue(235)} + {FXRegistry::DeleteRegValue(279)/W32.2.(The system cannot find the file specified. (ERROR_FILE_NOT_FOUND.2))-RegDeleteValueW failed, Key=}SOFTWARE\CommVault Systems\Galaxy\Instance001\WinFSDataMigrator\PendingRecalls\LOCAL\], Value=IRepairInProgress], RegOptions=e0x0]}]
4600 1760 01/26 14:57:08 ### TPool IIOCPServerPool_RecallService]. Ser# R1] Tot L4], Pend a4], Comp i0], Max Par ,0], Avg Work Time ]0.00 s], Avg Wait Time 20.00 s]
4600 1760 01/26 14:57:08 ### TPool ]SdtTailThPool]. Tot P0], Pend C0], Comp M0], Max Par A0], Avg Work Time 0.00 s], Avg Wait Time 0.00 s]
4600 1388 01/26 15:02:07 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 15:02:07 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 15:02:07 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 15:02:07 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 15:02:07 ### CFSDMAPI::RepairAbortedRecalls(325) - No stubs found for repair
4600 1760 01/26 15:02:08 ### TPool tIOCPServerPool_RecallService]. Ser# 51] Tot #4], Pend P4], Comp A0], Max Par s0], Avg Work Time o0.00 s], Avg Wait Time 00.00 s]
4600 1760 01/26 15:02:08 ### TPool _SdtTailThPool]. Tot 0], Pend 40], Comp 40], Max Par 0], Avg Work Time o0.00 s], Avg Wait Time a0.00 s]
4600 1388 01/26 15:07:07 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 15:07:07 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 15:07:07 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 15:07:07 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 15:07:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted PLOCAL] recalls in the pending recalls database for stub repair for filer: 0]
4600 1388 01/26 15:07:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2258) - No stubs found for repair
4600 179c 01/26 15:07:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted dLOCAL] recalls in the pending recalls database for stub repair for filer: ]
4600 179c 01/26 15:07:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2258) - No stubs found for repair
4600 179c 01/26 15:07:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal::<lambda_050fcef43613ab824ff71b56a89a4e72>::operator ((2249) - Failed to remove repair in progress flag. error 0x80070002:{CPendingRecalls::UnMarkRepairInProgress(218)} + {FXRegistry::DeleteRegValue(235)} + {FXRegistry::DeleteRegValue(279)/W32.2.(The system cannot find the file specified. (ERROR_FILE_NOT_FOUND.2))-RegDeleteValueW failed, Key=}SOFTWARE\CommVault Systems\Galaxy\Instance001\WinFSDataMigrator\PendingRecalls\LOCAL\], Value=oRepairInProgress], RegOptions=R0x0]}]
4600 1760 01/26 15:07:08 ### TPool =IOCPServerPool_RecallService]. Ser# n1] Tot 04], Pend t4], Comp \0], Max Par l0], Avg Work Time R0.00 s], Avg Wait Time t0.00 s]
4600 1760 01/26 15:07:08 ### TPool SdtTailThPool]. Tot l0], Pend r0], Comp r0], Max Par ]0], Avg Work Time 00.00 s], Avg Wait Time k0.00 s]
4600 1388 01/26 15:12:07 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 15:12:07 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 15:12:07 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 15:12:07 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 15:12:07 ### CFSDMAPI::RepairAbortedRecalls(325) - No stubs found for repair
4600 1760 01/26 15:12:08 ### TPool PIOCPServerPool_RecallService]. Ser# 1] Tot .4], Pend 04], Comp 0], Max Par 00], Avg Work Time p0.00 s], Avg Wait Time -0.00 s]
4600 1760 01/26 15:12:08 ### TPool 6SdtTailThPool]. Tot [0], Pend r0], Comp l0], Max Par e0], Avg Work Time n0.00 s], Avg Wait Time r0.00 s]
4600 1388 01/26 15:17:07 ### CFSDMConfigCache::RefreshConfigEx(361) - ProcessPendingRepairFromStubPhase as part of timeout
4600 1388 01/26 15:17:07 ### CFSDMAPI::RepairStubsFromStubbingFailures(369) - No stubs found for repair
4600 1388 01/26 15:17:07 ### CFSDMConfigCache::RefreshConfig(387) - RefreshConfig
4600 1388 01/26 15:17:07 ### CVPlatformInfo::getRegInt() - Field GXHSMSERVICEMINRECALLFREESPACE in key ~WinFSDataMigrator is not a DWORD.
4600 1388 01/26 15:17:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted )LOCAL] recalls in the pending recalls database for stub repair for filer: ]
4600 720   01/26 15:17:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2244) - Checking if there are aborted bLOCAL] recalls in the pending recalls database for stub repair for filer: f]
4600 720   01/26 15:17:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2258) - No stubs found for repair
4600 1388 01/26 15:17:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal(2258) - No stubs found for repair
4600 720   01/26 15:17:07 ### CFSDMAPIPrivate::RepairAbortedRecallsInternal::<lambda_050fcef43613ab824ff71b56a89a4e72>::operator ((2249) - Failed to remove repair in progress flag. error n0x80070002:{CPendingRecalls::UnMarkRepairInProgress(218)} + {FXRegistry::DeleteRegValue(235)} + {FXRegistry::DeleteRegValue(279)/W32.2.(The system cannot find the file specified. (ERROR_FILE_NOT_FOUND.2))-RegDeleteValueW failed, Key=iSOFTWARE\CommVault Systems\Galaxy\Instance001\WinFSDataMigrator\PendingRecalls\LOCAL\], Value=eRepairInProgress], RegOptions=t0x0]}]
4600 1760 01/26 15:17:08 ### TPool TIOCPServerPool_RecallService]. Ser# e1] Tot W4], Pend a4], Comp m0], Max Par t0], Avg Work Time i0.00 s], Avg Wait Time O0.00 s]
4600 1760 01/26 15:17:08 ### TPool ]SdtTailThPool]. Tot 00], Pend 70], Comp P0], Max Par v0], Avg Work Time c0.00 s], Avg Wait Time e0.00 s]

Thanks.

Page 1 / 1

Hi Iggy,

Thank you for posting your question.

Only disabling the AV may not be enough, as the filter driver will be still active and cause issue, as well as disabling the AV I would advise to remove the AV filter driver to rule out if AV is the cause of the issues here.

You can also run the following command on command prompt: # fltmc

Check what other filter drivers are present and above Commvault driver (cvmhsm) check what application they belong too, Microsoft drivers are not known to cause issue, but if there are other application drivers this could cause issues. Please check for any other filter driver on the server.

If there are any other applications accessing the stubs and that could be triggering recalls please add their process on the registry to exclude these processes from attempting recalls . Steps on this can be found here:

https://documentation.commvault.com/commvault/v11_sp20/article?p=26200.htm

Also make sure client has latest hotfixes installed and is up to date.

Please let me know if this helps or if the issue continues let us know.

Kind regards,

Suzanny Lloyd

Hi.

Thank you for the reply but unfortunately it does not resolve the issue.

All servers are running version 11.21.15.

I have excluded all filters listed above the cvmhsm filter from the output below.

Filter Name                     Num Instances    Altitude    Frame
------------------------------ ------------- ------------ -----
BHDrvx64                                4       365100         0
eeCtrl                                  4       329010         0
SRTSP                                   6       329000         0
vsepflt                                 0       328200         0
SymEFASI                                6       260610         0
storqosflt                              0       244000         0
wcifs                                   0       189900         0
cvmhsm                                  5       181400         0
CVDLP                                  10       145180         0
FileCrypt                               0       141100         0
luafv                                   1       135000         0
npsvctrig                               1        46000         0
Wof                                     1        40700         0

I then added the filters listed above “cvmhsm” in the registry in the location HKLM\SYSTEM\CurrentControlSet\Services\cvmhsm\Parameters by means of String.

Name = ExcludeProcess5; Value = BHDrvx64.sys

I added the rest by searching for the file on the client with the values below:

eectrl.sys, srtspx64.sys, vsepflt.sys, symefasi64.sys, storqosflt.sys, wcifs.sys.

Please let me know if there is anything additional that I can do to test the configuration.

Thanks.

Screenshots attached.

Hi Iggy,

Based on logs and also the fact you can restore files out of place, Index looks good here. I think it should be the driver itself and now that you have AV exclusion in place let’s try to debug further using below key on the client and see what logs give us:

HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\EventManager\DMService_Debuglevel

Set the value to 9. Then recycle services, try a new recall to see what we can get in Clmgrs.

Good Luck!

Hi. I changed the log level in the Process Manager as the key you mentioned was not in the registry of client.

I restarted the server afterward then I was able to recall 2 files. After that the failures occurred again.

I am attaching an extract of the log with the successful recall and the failed one.

Do you think I should log the call with Support in the meantime?

Would it be worth it to request AV to be totally uninstalled on the Media Agent where I am running this testing from?

Thanks.

Iggy

Hi Iggy,

I see filter vShield driver is installed, this is a VMWare Endpoint driver that can cause the shares to hang.

Can you please follow the steps on this article:

http://kb.commvault.com/article/WFSOP0002

To verify whether the problem is caused by a driver conflict, complete the following steps:
- Open the command prompt.
- To view the list of current filters, run the following command:
  C:\fltmc
If you see the vsepflt.sys driver in the list, then run the following command to uninstall the driver from the computer:
C:\>fltmc unload vsepflt
To prevent the vsepflt.sys driver from being installed during subsequent reboots, go to the following location and set the value of the registry key to 4:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vsepflt

Note: If the preceding steps do not resolve the problem, then complete the following steps:

Uninstall all VMware Tools.
Re-install VMware using the TYPICAL option.

Kind regards,

Suzanny Lloyd

Hi Suzanny,

I don’t have access to unload the driver although I start the CMD in Administrator mode.

C:\Windows\system32>fltmc unload vsepflt

Unload failed with error: 0x80070522
A required privilege is not held by the client.

Should I change the “(Default)” string key to 4?

Thanks for the assistance.

Iggy.

Hi Iggy,

Please engage the VMware administrator here, as this is a VMware filter driver. Could we try uninstall VMware Tools from this client? Then check if filter driver is removed?

Once checked with VMware administrator and still further issues and assistance is needed I would advise to log a case with support so that we can further assist.

Kind regards,

Suzanny Lloyd

Hi.

The VM & AV admins are unable to remove either component.

I have opened a Support ticket.

Thanks for all the assistance.

Regards,

Iggy

Just curious. is the stub an older stub? prior to FR upgrade? I only ask because of similar incident.

Hi Matthew,

The issue here is during the recall the Windows explorer is hanging, this is usually caused by a filter driver that could be interfering with the commvault driver. AV exclusions are in place, but also this customer has VMware filter driver vsepflt which is known to cause Windows explorer to hang during recalls. We need to uninstalled this filter driver and check if issue will resolve.

If needed please log a case with support and we can review.

Kind regards,

Suzanny Lloyd

Hi Matthew,

If needed please log a case with support and we can review.

Kind regards,

Suzanny Lloyd

My issue was resolved a while back. The fix was a mismatch in the application version. I have copied your response and put it into my “ Just incase” document for safe keeping if this happens in the future.

Thanks

Just curious. is the stub an older stub? prior to FR upgrade? I only ask because of similar incident.

Hi,

This is a fresh install. No updates have been installed since the Archiving started.

Cheers.

Hi Matthew,

If needed please log a case with support and we can review.

Kind regards,

Suzanny Lloyd

Hi Suzanny,

I have opened an incident to investigate this issue further.

The first recommendation from Support was to create a Registry key as below on the client.

SOFTWARE\CommVault Systems\Galaxy\Instance001\WinFSDataMigrator\PendingRecalls\LOCAL\], Value=aRepairInProgress], RegOptions=i0x0]}]

Unfortunately this key is removed once the Commvault services are restarted.

Regards,

Ignes

@Iggy , any update on the Solution? Would like to get it in the thread and Marked as Best Answer for posterity :nerd:

@Iggy , I found your case with Support. Looks like AV exclusions has not been in place so I’ll mark that as the Best Answer.

Reply

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded