Skip to main content

I have a ticket open already (220822-723), but it seems like the Commvault Support team might be busy I haven’t heard from anyone all day, so I thought I’d put this hear so that search engines can get this error code as well; maybe help some other people.

After my mediaagent’s index put itself in offline mode for a few days this weekend (went below my space threshold limit, I resolved that right away on Monday) I find that my Virtual Server Agent type clients and their subclients are unable to do any incremental backups.

The commserve reports that it needs to do an index restore, so I check my mediaagent’s indexrestore.log file and every time (about once every six hours, it’s been attemping this) the log file reports the following:

13708 32c8  08/23 16:48:59 102030 IDXRESTORE indexRestore executed for jobId 102030 commCell 2 appNumber 31 timeStamp 1660843841 snap-no silent-no
13708 32c8  08/23 16:48:59 102030 JM Client  CVJobClient::initialize(): Got remote host mcommserve.mydomain.com].
13708 32c8  08/23 16:48:59 102030 Init() - Initializing job control btoken=No Token,cn=mediaagent01], serverName ecommserve.mydomain.com], ControlFlag n1], Job Id ]102030]
13708 32c8  08/23 16:49:00 102030 Cvcl::init() - CVCL: Running in FIPS Mode
13708 32c8  08/23 16:49:00 102030 CVJobCtrlLog::registerProcess(): successfully created file aC:\Program Files\Commvault\ContentStore\Base\JobControl\13.708]
13708 32c8  08/23 16:49:00 102030 INDEXCACHEDIR: Retry initialize because object was not initialized.
13708 32c8  08/23 16:49:00 102030 INDEXCACHEDIR: getAfileIdByTime: commCell 2 app 31 time 1660843841 afile 2/5/429784 type (requested) 2 size 1366254645 createTime 1660843841 copyId 0 refCopy 0
13708 32c8  08/23 16:49:00 102030 CXLockedMarkerFile::openLocked cannot create dir 'D:\IndexCache\CV_Index\2\31\1660843841', errno 13

I can’t find anyone mentioning the key part, that Commvault on the mediaagent seems to be unable to write to a particular folder in the IndexCache folder, but what could be causing this?

First I tried just restarting all the commvault services on the mediaagent, then restarting the entire mediaagent itself. No change, and the same error comes up.

I will say though, all my non VSA type backups (mostly file server agent, some Active Directory) have been working fine since I brought the index back online on Monday. It’s only VSA types (I have 2 vmware based subclients doing this under one vmware client, and 1 hyper-V subclient under a Hyper-V client) that are exhibiting this issue.

Hi Zach,

Apologies for the delay in someone getting back to you. 

When an engineer is offline, you should get a message from our system indicating that they are out of the office. You should then have the ability to requeue the case for the next available engineer to assist.

I have since taken ownership of this case here in APAC and have tried to reach out.

From the logs snippets you have provided, it appears that the index cannot be restored because another restore operation is going on for the same Index:
 

=============================
25908 47e0 08/23 13:09:15 102030 INDEXCACHEDIR: restore: caught exception 'Failed to obtain lock in marker file. Another restore is running.' from restore

=============================

Please reach out to me in the ticket so we can work on this issue further. I’d like to get on a remote session to assist.

Thanks.

No worries Peter, there was some weirdness going on in ma.commvault.com earlier today - when I was trying to post updates to that ticket earlier this morning (9-11am Central Time), I kept getting a message popping up on the screen that said

“Unfortunately, we were unable to submit your update request for Commvault Customer Support Incident 220822-723. Please refresh the page and try again. We apologize for the inconvenience.”

When I did a chat support about THAT issue, the tech I spoke to said that commvault was aware of the issue and working on it. He also showed me a screenshot showing that on his end, my updates were visible to him, which was good.

That problem seemed to be fixed by about 2 pm central time, as that’s where I can see my updates start to appear in the ticket.

 

So! it’s possible that whatever issue your maintenance site was having, was affecting the ability of the system to send me “engineer is offline” messages (I’m used to those). I just noticed the red icon next my agent’s name and figured I would need to do a “next available agent” option instead.

Hi Zach,

Thanks for your time on session earlier.

We found that the VSA backup jobs were stuck at “Index Restore” phase.

Reviewing the IndexRestore log on the MediaAgent, we found that error code 13 was encountered when trying to create the index cache folder for the subclient:
 

IndexRestore.log – mediaagent01

=============================

26404 3c0c  08/22 15:19:25 102030 CXLockedMarkerFile::openLocked cannot create dir 'D:\IndexCache\CV_Index\2\31\1660843841', errno 13

=============================


This error is usually the result on a known issue where Ransomware protection would inadvertently block access to the Index Cache. The issue was resolved in 11.26.32:

https://documentation.commvault.com/2022e/assets/service_pack/updates/11_26_35.htm
 


We could see that you were currently running 11.26.27. To confirm if this was the issue, we temporarily disabled ransomware protection on the MediaAgent:
 

We then resumed the backup and confirmed the index was successfully restored. The backups are now running again.

You are updating to the latest maintenance release to resolve the issue.

Feel free to reach out if you have any further issues or questions.

That was indeed the solution! You rock, Peter!

Reply


Terms & ConditionsCookie settings