Workflow to resubmit failed job with particular failure reason

  • 27 May 2021
  • 8 replies
  • 353 views

Userlevel 1
Badge +6

Hi CVcommunity,

 

I do have a question and maybe some of you could point to what first steps should be taken to perform the following:

 

  1. We have backup jobs tha are failing with error codes such as ‘Waiting for services to come online’, ‘Error encountered when closing virtual device. Please see SQL server vdi.log for more details’ and etc. - Most of those jobs usually are fixed by simply resubmitting the job or resubmitting it after restarting client services….

Is there an option that Commcell could run a workflow itself when job fails with such failure reason?

Workflow should restart client services and resubmit the job.

 

 


8 replies

Userlevel 7
Badge +21

Hey @Benjis , I’m seeing if anyone internally can respond, and I changed this to a conversation to allow for others to share their own solutions.

Userlevel 1
Badge +2

Hi @Benjis if the job pending reason is “Waiting for services to come online” then a workflow also will not be able to restart services remotely because most likely the main communication service is down.

 

for the second case, one option is to create a custom alert rule that monitors for that condition and then kick of a script or workflow to take the corrective action. This is how you can create a custom alert rule using a sql query. I think there is a RunningBackupJobs view that can be used in your sql query

https://documentation.commvault.com/commvault/v11_sp20/article?p=5308.htm

Userlevel 1
Badge +6

Hi @Amey Karandikar 

 

I’ve checked your suggestion and it seems there is already some kind of alert rule which might be the thing we are looking for, but I am not the guy who understands sql queries.
 

Alert rule

 
SQL query

Would someone be able to “decrypt” that SQL query and tell what it is it? :D

 

 

Userlevel 1
Badge +2

Hi @Benjis 

that one seems to be specific to jobs that are already finished. i believe you are looking to fix the problem while the job is active but in a pending state. You can do a query like this:

 

select jobid, clientDisplayName, DelayReason  from runningbackups

 

and add the where clause for the Delayreason column to match the specific error you want to fire the alert for and then when alert is triggered you run the workflow 

Userlevel 1
Badge +6

Hi @Amey Karandikar

 

This particular error: ‘Error encountered when closing virtual device. Please see SQL server vdi.log for more details’ - when job get’s this error - it continues after automatic resume (job manager settings) - I want to have a workflow to be triggered once job will actually fail - and when it fails it comes with status ‘Completed w/ one or more errors’ and failure reason is the one above. So I’d like to have a workflow kick in once such job fails.

Userlevel 1
Badge +2

Hi @Benjis 

You can setup a job failed alert and in the alert setup a filter for the error code or failure reason so that it fires only when that error happens. Then setup a workflow to run when the alert is triggered.

Userlevel 1
Badge +6

Hi @Amey Karandikar 

 

Is there some sample workflows that are already made to resubmit jobs? I don’t have experience with workflows and would not know how to create one at this point.

Userlevel 1
Badge +2

@Benjis 

i dont believe there is an exact sample for resubmitting jobs. There are however lot of workflows on the store that you could use as a reference point. I think you will need to configure a workflow with job id as input from the alert and then execute a rest api or qoperation command line to resubmit the job.

 

Reply