Solved

Synthetic Full not running per schedule


Userlevel 2
Badge +8

(split from a previous thread)

Thanks Mike I figured not possible as needing to chop that chain (first inc after sync synchronises etc).

We’ll ned to keep the incs for the required retention and see if we can get synthetics running properly. Job summary only shows incs so no reasons for synth not running. However, what I see in job history for client is the synth schedule does kick in and runs its incremental (before) backup but then no synthetic full gets run.

icon

Best answer by dude 4 March 2021, 19:00

View original

17 replies

Badge +15

@Mike London UK If you pick one of these clients that are supposed to run a synthetic full job, right click on it, view then schedules. 

Right click on the Synthetic Full schedule and View Jobs, do you see jobs that errored or something else? 

 

 

Userlevel 7
Badge +23

 

We’ll ned to keep the incs for the required retention and see if we can get synthetics running properly. Job summary only shows incs so no reasons for synth not running. However, what I see in job history for client is the synth schedule does kick in and runs its incremental (before) backup but then no synthetic full gets run.

 

Really strange. Does it work if you run it manually? - that way you have a job ID and can eye out any issues, or whittle it down to a scheduling problem - you can always cancel it (its a Media Agent only operation so wont interrupt the VMs).

Userlevel 7
Badge +23

@Mike London UK my initial thought on the report is that you didn’t include failed jobs and details, though what everyone else is saying is accurate.

Check the Job History (include failed/aged jobs) and see if the Synth shows up.

If you have the schedule set to run an Inc before the Synth starts, is it possible that Inc is still in the job controller?  I’ve seen that before as well.

Userlevel 2
Badge +8

Running a manual synthetic works (as expected). There are no errors when doing the job history, either from schedule or subclient. The schedule is actually running the incremental backup (job details shows the schedule name) but the synthetic just doesn’t appear. @Mike Struening if you’ve seen that before what was the fix? Such difficult thing to catch unless watching jobs very closely (most people are watching for errors).

Userlevel 7
Badge +23

The fix is to set an option in Job Management called Queue jobs if other conflicting jobs are active, though it will affect ALL jobs.

What this setting does is queue jobs that attempt to run when another job for that subclient is already running.  This way, if your Synth full tries to run when a previous incremental is not finished yet, it will queue up in the job controller and wait until the Inc finishes.

If this is your issue, take a look at the scheduled time for the Synth as well.  Might just be starting too soon (compare it to the time the Incs take).

It is definitely easy to miss.  Would suggest scheduling a weekly job summary report with the failure reason included.

There’s also an option for how long you keep job history details called Days to keep the failed/killed backup job and other job histories, though I can’t imagine you haven’t had a Synth Full in the past 90 days…

When you look at the Job History and see the Synth full that never completed, is there any error in the Job Details?

Badge +15

The fix is to set an option in Job Management called Queue jobs if other conflicting jobs are active, though it will affect ALL jobs.

What this setting does is queue jobs that attempt to run when another job for that subclient is already running.  This way, if your Synth full tries to run when a previous incremental is not finished yet, it will queue up in the job controller and wait until the Inc finishes.

 

Great addition there @Mike Struening . What makes me scratches my head is that from his description it seems that the Synthetic Full isnt even been started. 

From what I have seen in the past, if the incremental is still running and the synthetic schedule kicks in, the job will most likely fail with the “Failed to Start: Another backup is running for client [...], iDataAgent [...], Instance [...], Subclient [...].”

I would double check the schedule policy and make sure that the selection within the schedule is indeed Synthetic Full. 

Make sure that the Day selected for the synthetic full is not conflicting with the incrementals. Meaning, look at the last incremental before the actual day where the Synthetic Full is supposed to run and see how long that incremental run for. If actually finished before the synthetic was supposed to start or if it ran over the schedule for the synthetic full.

 

 

 

Userlevel 7
Badge +23

Yeah @dude , I have the same confusion….something is not right.  Either the schedule never actually ran, the failed job details pruned off, or the viewing of the jobs is not including the aged jobs/failed jobs.

@Mike London UK , would you mind copying a screenshot of the Job history for this subclient?  Make sure you include Aged/Failed jobs (and blur out the client names).

If you DO see the Synth full, send a screenshot of the job details/errors/failure reasons.

Userlevel 2
Badge +8
Here’s there screenshot. Synth run on Saturdays, so Feb 27 and back.
 

Something definitely isn’t right :-) When you look at details for the incs run by synth schedule you see

The top synth has been run manually and we’ll be watching the backup this Saturday closely.

Badge +15

As much as that looks like to be a Synthetic Full Schedule, that only represents the name the schedule was given.

Can you please;

Locate Schedule Policies→ Locate that Synthetic Full Schedule → Right Click on it and Edit and Make sure you actually see a Tasks with the Job Type Synthetic Full as the screen shot below shows?

 

Once you are in the Schedule Synthetic Full → Schedule Pattern → Options → Make sure that the Advanced Settings says “Never End or there is no End date” 

 

Userlevel 2
Badge +8

What we have in the schedule policy is:

The 1832 start use to be 1830 but we’ve found some instances where incrementals weren’t’ running and just changing the time slightly fixed the issue. I’m sure this isn’t the case here as we are seeing the schedule run but just not going beyond the incremental. 

And in the synth schedule we have:

We’ll move to a full blown support ticket after this Saturday if still a problem. We also have to go and check all the other schedules as well just to see how widespread the issue is.

Thanks for your replies so far.

Badge +15

So here is what I would suggest. 

You incrementals are running at 18:30 and on average are taking 30 min to complete. 

However your Synthetic Full is scheduled to start at 18:32 and that to me is a conflict on itself. I believe this is one of the issues here.

 

Can you move the Synthetic to like 19:30? That would give enough time for the Incrementals to finish (again average 30min). 

One other thing, Im not sure why you are running the incremental before the Synthetic Full, Id just uncheck that and let it use the schedule for the incrementals. Thats why you have them there.

Userlevel 2
Badge +8

Hi @dude it’s not a conflict as the normal incremental schedule is excluded from running on all Saturdays. The problem with having a constant incremental schedule (CV’s default these days) is that you cannot predict finish times and if one takes longer than the gap to the synthetic then we have a failure. Forcing jobs to queue has to be done at too higher a level. It would be nice for synthetics to have a config option to queue until any existing job is complete.

Badge +15

Hi @dude it’s not a conflict as the normal incremental schedule is excluded from running on all Saturdays. 

I would agree to that if the screen shot you shared didnt show a job starting on the 20th and 27th exactly around at 18:30 which is the same time as the incrementals. It almost seems that the schedule is ignoring the exceptions you added as part of the Schedule Pattern for the Incrementals. 

If you can to incremental, could you change it to Weekly, make sure that all week days are selected except Saturday? I use the exact same setup and no issues with that.

CV may or (probably not) be understanding the exception you added as part of the Daily.

 

Userlevel 2
Badge +8

I think it’s more subtle than that and that CV is confusing the incremental and synthetic full jobs (as the Sat inc job details shows the synth schedule). There’s now a few minutes separating the schedules so we’ll see what happens this Saturday.

Userlevel 7
Badge +23

Man, you guys found so much in such a short time!

As the @dude already pointed out, the Inc is definitely running on those Saturdays which was my initial suspicion.

Check your schedule on those Incrementals and set Weekly instead, then enable each of the days you want (i.e. don’t check off Saturday).  I agree that it doesn’t seem to be acknowledging the multiple exceptions (or more likely, there’s something more complex happening).

Also, here’s that Queue Job option screenshot, which would resolve the issue, but is a Commcell wide change.  The schedule adjustment is a much more precise fix.

 

Userlevel 2
Badge +8

We have the jobs running as expected now. Basically, we shifted the synthetic start time by 5 minutes (after the incremental) and deleted the exclusions on the incremental schedule and added them back.

Thanks for all the responses.

Userlevel 7
Badge +23

Glad to ehar it, @Mike London UK !  Hopefully everything else mentioned here will aid you at some point in the future as well :nerd:

Reply