sucessful backups not protecting files The process cannot access the file because it is being used by another process

@Onno van den Berg Thanks for the info.

Is that also valid for folders? And in what FR was it introduced, do you know?

//Henke

I did try forcing this using the key however it still failed.  will review the logs further.

okay so I investigated it further; although the VSS writers were showing as okay; the VSS providers had an additional provider Acronis VSS sw provider

and VSS errors on the event log 

12292 during backup

Checking with customer now if they have any other backup agent installed and shared the following article for them to review.  

“

Hello guitarfish,

Thank you for posting this information in our forum.

Did you check if this, after upgrading from Acronis Backup & Recovery 11 to Acronis Backup & Recovery 11.5?

1. Go to Start -> Run -> regedit.exe and press ENTER;
2. Navigate to this key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\Providers\
3. Check, if you can find at the left part entries below Providers. There should be only one entry.

   Under every condition: Click on every entry and check in the right part, if you can find this entry: "Microsoft Software Shadow Copy provider 1.0". Do not delete this entry!

   

   

   For the other entries: Check if you can find the Acronis VSS Provider entry and delete it.

4. Restart the machine.

“

yes!!   

In regards to the reprocessing:   during file scan we build a collect file.  That collect file is passed to the backup process to protect the files.  What is happening is inbetween the scan and backup, the files disappeared.  The backup will flag that as a specific failure with an appropriate context/error code.  In the screenshot below we flag windows error code 3 (first red box) for path not found (it was in the scan, but when we went to back it up it wasnt there, and the second red box was error code 2, the file is not there.  

In those cases remediation isnt possible, but for other conditions we should explore a retry (I’m almost positive we try again at the end of phase runtime, but ill check with dev)

We do this today in several areas.  

We highlight the last backup status and depending on the failure type we provide some quick recommended actions:

You can see there is a difference between the failed to start recommendations and the failed VSS snapshot.  When we first started talking about this, this is where I was thinking we can inject a workflow/action that stops and restarts VSS (or something like that.

 Here is from VM group where you can resubmit and  backup just the failed VMs:

Here is a shot from databases:

Well, @MFasulo is the right guy for Command Center, no question.

Can you link me to your idea about the post phase scan?  I want to ensure it’s getting traction/attention.

@MFasulo , File Scan does provide a list (logs below), but I’m not aware of any actual remediation; only status reporting.

@Onno van den Berg , in your vision, would the job pause and wait for admin action/corrective action or simply provide more upfront detail?

Here is an output from a lab backup I just ran to test:

2640  5f4   02/09 08:42:30 165759 CsSnapRequestor::AddWritersToVolumeList() - Added writer(s) <[Task Scheduler Writer, d61d61c8-d73a-4eee-8cdd-f6f9786b7124] [VSS Metadata Store Writer, 75dfb225-e2e4-4d39-9ac9-ffaff65ddf06] [Performance Counters Writer, 0bada1de-01a9-4625-8278-69e735f39dd2] [System Writer, e8132975-6f93-4464-a53e-1050253ae220] [ASR Writer, be000cbe-11fe-4426-9c58-531aa6355fc4] [Registry Writer, afbab4a2-367d-4d15-a586-71dbb18f8485] [BITS Writer, 4969d978-be47-48b0-b100-f328f07ac1e0] [WMI Writer, a6ad56c2-b509-4e6c-bb19-49d8f43532f0] [COM+ REGDB Writer, 542da469-d3e1-473c-9f4f-7847f01fc64f] > to included writer list.

2640  5f4   02/09 08:42:30 165759 CsSnapRequestor::StartSnapshotSet() - Created shadow set 6a1137ec-d491-4d73-9b27-547c64bea85e

2640  5f4   02/09 08:42:30 165759 CsSnapRequestor::AddVolumesToSnapshotSet() - Successfully added volume [C:\] to shadow set.

I agree.  @Mike Struening   do you know if we do any VSS writer remediation as part of some error output?  I recall back in my support days we did post the VSS writer status before backup and after backup, not sure if we still do that.  

 

Yes you are right there are many jobs that completes with errors, though we thought that missing one file is bad enough. So we changed the setting, I think it’s this that we talk about, to 0 failed files.

This gives an indication to what clients needs attention, and most of them can be dealt with applying filters, either global or local. Some can’t be filtered out though, such as SystemState and so on.

Most of the problematic ones are systems with alot of “temp” files, that seem to be flagged for backup but when the backup occurs they aren’t there, hence are failed due to the system.

 

In addition we create alerts for systems under compliance audits for failed files/jobs.

 

 

 

@Henke what Commvault could introduce is a post process that process the list of failed files and performs another file scan to see if the files still exist. That could filter out the temp files automatically and could reduce the amount of reported files. The issue with Temp files being wiped while the job is progressing will also be more visible with long-running jobs.

maybe this is something to look into!

My .02 on this:

IF the writer is in a bad state, that needs to get corrected, before anything.   

If the writer still fails on that database file, I would suggest looking into how that vendor recommends protecting the file.   If there are freeze/thaw scripts or quiesce scripts, those should be applied to the subclient.  This can be done in command center.  

I agree with this Henke… which is why only under extremely rare cases would I personally suggest manipulating how jobs are classified, based on errors. 

To me, errors and failed files, mean “fix me”.  If those are files you dont need to protect (like tmp/cache files),  we can filter them.   In command center, filtering can be done globally (manage > system global filters), on server group level (configuration tab > file exceptions), on a plan level (through backup content settings), and on the subclient level (through custom backup content)

Like SLA, if its not 100% something is wrong and needs corrective action.  

That’s a clever idea .  Let me pass that up the chain and see what I can do!

Thanks, @Stefan Vollrath !

Good conversation here!  the main (historical) reason for job completion status was that it can be normal for a few files to fail for various reasons.  If you marked every job that skipped one file as Completed with Errors, they’d all look that way.

Completed with Errors is reserved for either system State Components getting missed, or expected Databases getting missed (there might be others escaping me now).  The status really means “we ran and finished, but something major was missed; we advise looking” as well as “we won’t prune previous Completed Without Errors jobs until this is fixed” by default.  There’s some extra logic that goes into play for CwE.

For your request @Onno van den Berg for Command Center inclusion, I’ll get the right people to respond.

For the OP, @Theseeker , let us know if you’re able to address the VSS issues via a reboot and if subsequent backups work as expected.