Question

Workflow Script API

  • 10 February 2021
  • 13 replies
  • 167 views

Userlevel 1
Badge +3

Documentation for Workflows mentions some Functions for Java to interact with or access other Resources during execution https://documentation.commvault.com/commvault/v11_sp20/article?p=49729.htm.

But alone from browsing through the other Pages it gets clear there are way more options like activity.exitCode or workflow.setFailed().

Dissecting built in Workflows reveals other treasures like csdb.execute(“sqlquery”).

Is there a complete list of all available Functions and how/when to use them? Trial-and-Error and Reverse-Engineering by Customers seems error-prone and inefficient.


13 replies

Userlevel 4
Badge +11

Hey @Stefan Vollrath!  Have you checked out the Activities list?

https://documentation.commvault.com/11.22/expert/49662_workflow_activities.html

Sounds like you’re looking for ALL possible activities in one place (like a referral list)?  Is that right?

Let me know how you’d like this to be arranged and I can add in the Documentation folks :nerd:

Userlevel 1
Badge +3

Hi @Mike Struening,

I know the Drag-And-Drop Activities, tried most of them. Most are rather limited in ability (f.E. Create-Subclient cant manage Excludes or FS-Options) or even outdated (f.E. Operation-Backup being unable to run HANA DB Backup).

Ultimately I end up using mostly CommServeDBQuery to retrieve Data from DB and Execute Blocks to run xml-Requests, sprinkling of some helpers like ExecuteScript, LogEvent or ResultSetToText, glued together with Conditional Connector or Script-Blocks.

The latter ones are what I am interested in. It’s way more flexible that way, reading Input-Values or Result-Sets, creating new Strings depending on complex conditions and writing them out to Variables for other Actvity-Blocks to use. Or to catch Error-Conditions that Execute-Block doesn’t catch itself and diverge execution path based on that. Trying to accomplish that with Switch/Desicion/AssignValues Blocks gets messy fast.

Don’t intent to use it as excessive as Devs did in “End User Automation” Workflow. But that one is a good example on how rich on advanced Options the Workflow-Engine is, above and beyond the Lego-Block simplicity.

Hoped the Lego-Stuff would be part of the Essential-Docu and those Scripting-APIs be expanded upon in Expert, maybe that will be Part of the “Hardcore”- or “Admin”-Sections :-D

Cheers,

Stefan

Userlevel 2
Badge +1

Hello @Stefan Vollrath 

Just wanted to see if using the HTTPClient (to call APIs) will help reduce the complexity for you? 

CommServeDBQuery mean you have to know the DB very well and I have seen some very deep knowledge of flag codes etc to really guarantee you are getting what you expect in to the results.

Take a look at the https://api.commvault.com and use postman and API calls you need and then just add them in to the HTTPClient in the workflow passing  the results from one to the next as needed.

Hoped the Lego-Stuff would be part of the Essential-Docu and those Scripting-APIs be expanded upon in Expert, maybe that will be Part of the “Hardcore”- or “Admin”-Sections :-D

I know what you mean on this bit. We do really rely on some of the advanced knowledge to be understood without teaching it. Knowing Java is definitely useful.

Feedback like this is great, did you know you can post feedback in documentation? It might be worth doing that as well as it will trigger a review in the background that may end up giving you what you want for future documentation versions.

Userlevel 1
Badge +3

Hi @Graham Swift 

API is in some parts even worse if I’m honest:sweat_smile: Already using it to configure Things that weren’t available as QCommand-Activities or that do not have xml-Requests documented, like setting OperationWindows. It is still super limited in FeatureSet (as of FR20). Intended to use it for Automation to register and configure new Clients back in SP14, but half of the Operations are not available and for the rest you need a university degree to collect all the needed infos from two dozen concated api calls. So Workflow we went. Not much different now. Yeah we don’t use Plans yet and still wait for them to get usable :zipper_mouth:

Documentation only shows you one or two Samples with carefully selected Attributes, but rarely explains what they mean, which are mandatory or optional and what Values are available. Having different ID-System in some places was also annoying, though that seems to be mostly aligned in fr20 now.

In general I don’t like to mix so many different Attack-Vectors. CVDB, XML-Requests and Java-Code are already plenty to have issues with. Using HTTPClient Requests and adding overhead logic to check the replies would bloat Workflow Screen quite a bit. Makes more sense using them in fully custom written API-Client.

Anyhow, thanks for the hint!

Userlevel 4
Badge +11

@Seema Ghai , is there anything on the roadmap for a more robust Workflow API section in our Expert section?

Userlevel 1
Badge +1

Hi @Stefan Vollrath, would adding more details about the attributes and more example help? What would make it easy and more usable? 

@Mike Struening, We will definitely evaluate the need for more information and add it to our roadmap.

Userlevel 1
Badge +1

Hi Stefan

Your comments so far, indicate you've done a fair bit with the workflow engine, so I thought I'd try and give you a bit of a tips & tricks braindump.

It's not particularly structured, but hopefully there's some nuggets of useful information in there for you, please don't hesitate to call out if you have a further specific question.

Firstly as you  say, it can be a lot easier to conduct some things in Java (Groovy), or Javascript rather than laying all logic out as activities and transitions, over time you get a natural feel for what is appropriate where and you seem to be well on your way to forming your own style of when to do what.

There are a number of libraries exposed in both Java and Javascript (and the ability to add additional Java libraries to the Workflow Engine Classpath). In addition as of FR21, The old Beanshell 1.x (loosely typed scripting subset of Java) has been swapped out for Groovy which is both compatible, and includes a lot more of the syntactic Java sugar (e.g. varargs), that is needed to easily consume a number of Primary and 3rd party Java Class Libraries/Methods, so including your own favourites is now a more practical and palatable possibility.

Javascript has changed also as of FR21, still Nashorn engine (a Javascript engine in the JDK), but this seems to support more es6 behaviours (at the cost of some of the Mozilla specific extensions in the previous version have also gone away).

Below is a bunch of information, You can consider a lot of this opinion, but my team works with large and complex workflows a lot, so a lot of this opinion, is hopefully, well formed :) 

As you've highlighted, a lot of interacting with Commvault, is building and parsing XML or JSON and taking actions. Below is my interpretation of the Commvault Engine from an automation perspective at the various layers 

There are the CV provided RESTFul API's and then the layer below that of submitting XML message fragments to the CTE via Operation Execute directly and processing the response appropriately. (e.g)

Automation Stack
​​

 

With that in mind most of my tips are around the processing of JSON and XML

TIP 1 - Don't forget Javascript. JSON is Javascript so often if you're processing JSON, it's a simpler journey using JS than it is using Java (Groovy).

(e.g. Javascript script to construct JSON object is very simple)

var obj = {};
obj.FailureReason = workflow.getLocalVariable("failureReason").trim();
obj.Status = workflow.getLocalVariable("status").trim();
JSON.stringify(obj);

TIP 2 - That said there is a JSON parsing library included in Commvaults default WorkflowEngine Classpath

import org.json.JSONArray;
import org.json.JSONObject;
import org.json.JSONTokener;

JSONTokener tokener = new JSONTokener(xpath:{/workflow/ForEachJson_1/value});
JSONObject row = new JSONObject(tokener);

String clientName = row.getString("ClientName");
String datacenter = row.getString("Datacenter");
String serviceLevel = row.getString("ServiceLevel");
String agent = row.getString("Agent");
String osType = row.getString("ClientType");
String sys_guid = row.getString("sys_guid");
Integer sys_id = row.getInt("sys_id");
Boolean hasPackage = row.getBoolean("PackageInstalled");
String status = row.getString("ConfigurationStatus");

// Set the default failure reason
workflow.setLocalVariable("sys_id",sys_id.toString());
workflow.setLocalVariable("status","Failed");
workflow.setLocalVariable("failureReason","Unhandled agent type ["+agent+"].");

// Fail if the package is not installed or configuration is already marked as completed
if(!hasPackage){
    workflow.setLocalVariable("status","Failed");
    workflow.setLocalVariable("failureReason","The ["+agent+"] package is not installed. Please install it before trying to configure it.");
    return "skip";
} else if(status=="Completed"){
    workflow.setLocalVariable("status","Completed");
    workflow.setLocalVariable("failureReason","Configuration is already marked as completed. No changes made.");
    return "skip";    
}

// Set the local variable values
workflow.setLocalVariable("clientName",clientName);
workflow.setLocalVariable("datacenter",datacenter);
workflow.setLocalVariable("serviceLevel",serviceLevel);
workflow.setLocalVariable("agent",agent);
workflow.setLocalVariable("osType",osType);
workflow.setLocalVariable("sys_guid",sys_guid);

return agent;

TIP 3 - If you want to work from JSON to XML - convert

 

JSON > XML

TIP 4 workflow.setLocalVariable and workflow.getLocalVariable

Activities are mostly self contained, no persistence other than referring back to the activity, or using globals such as workflow variables. Often workflow.setLocalVariable is a more consumable/concise choice for preserving state


Protip: You can refer to local variables  outside of your current (stack) frame of reference (e.g. in the below case this is a processblock within a processblock refering to a localVariable set in the parent Processblock)

workflow.getParent().getParent().setLocalVariable("outputStatus",1 ); 

local variables are consumable everywhere they are in scope (or out of scope using something like the above mthoed (e.g) expression scripts (covered later) 

using localvariable in expression

 

Or as inputs etc…
 

local variables as inputs

 

TIP 5 - workflow.setFailed

Hopefully this snippet makes this self explanatory

JS:

try{    
       var tableName = xpath:{/workflow/ProcessBlock_GetTableViewId/tableName};
       var tableId;
       var j = JSON.parse(xpath:{/workflow/HttpClient_GetTableId/output});
       for (var i = 0; i < j.length; i++){
           if(j[i].name === tableName){
               tableId = j[i].sys_id;
               break;
           }
       }
       if(tableId == null){
           msg = "Failed to find the id for the table ["+tableName+"].";
           logger.info(msg);
           workflow.setFailed(msg);
       } else {
           tableId; // return the table id
       }
   } catch (e) {
       msg = "Failed to find the id for the table ["+tableName+"] with exception ["+e.message+"]";
       logger.info(msg);
       workflow.setFailed(msg);
   }

 Java (Groovy) - Context verifying that a JSON dataset provided by a row in a custom report that a workflow is taking as input, has actually been provided.
 

// Check the selectedRows input is provided
if(xpath:{/workflow/inputs/selectedRows}==null || xpath:{/workflow/inputs/selectedRows}.equals("") ){
    workflow.setFailed("The selectedRows input is required but was null or empty.");
    return;
}

 

Bringing it all together (e.g. in the below screenshop validating the outcome of a HTTPClient activity replete with appropriate error-handling, messaging, exiting)

 

TIP 6 - Expression Types

 

 

TIP 7 (IMO the best tip :grinning: ) - Programmatically assembling and changing XML payloads rather than always utilizing a skeleton with xpath interpolation

Given you've worked with Workflow a bit, you'll be used to taking an XML stencil fragment and altering it with various interpolation of variables (e.g via xpath:{/foo/bar}

Often it can be more compact/readable to

  • Assemble an XML Programatically
  • Deconstruct a XML stencil fragment and modify/add/remove from it Programatically

There is a Java WirkflowElement class that comes in handy here

 

There are many options for parsing XML programatically or using activities (e.g)

 

And some programmatic examples

JAVA: Creating an XML payload (Context adding a additional setting to the client properties of a client)

// Build the request
XML req = utils.parseXml("<App_SetClientPropertiesRequest />");
//App_GetClientPropertiesResponse/clientProperties/clientProps/registryKeys
prop = req.addChild("clientProperties").addChild("clientProps");
// set the additional setting
rKey = prop.addChild("registryKeys");
rKey.setAttribute("keyName","sDb2ThresholdALFN");
rKey.setAttribute("relativepath","Db2Agent");
rKey.setAttribute("type","STRING");
rKey.setAttribute("value",xpath:{/workflow/configuration/sDb2ThresholdALFN});
rKey.setAttribute("comment","Added by workflow ["+xpath:{/workflow/system/workflow/workflowName}+"] at ["+xpath:{/workflow/system/startTime}+"]");
rKey.setAttribute("enabled","1");
return req.serialize();

JAVA: Altering an existing payload (context, this is referring to the output iterating through the output from a previous GET /InstanceProperties from DB2 instance in variable xpath:{/workflow/ForEachXml_DB2Instance/values})

// Get the instance properties
XML instanceProps = utils.parseXml(xpath:{/workflow/ForEachXml_DB2Instance/values}.toString()); 
logger.info(instanceProps);

String clientId = instanceProps.selectSingleNode("//values/@clientId").getValue();
String applicationId = instanceProps.selectSingleNode("//values/@applicationId").getValue();
String instanceId = instanceProps.selectSingleNode("//values/@instanceId").getValue(); //roger - values was instance
String instanceName = instanceProps.selectSingleNode("//values/@instanceName").getValue(); //roger values was instance
workflow.setLocalVariable("instanceId",instanceId);
workflow.setLocalVariable("instanceName",instanceName);

// Build the request
XML req = utils.parseXml("<App_UpdateInstancePropertiesRequest />");
prop = req.addChild("instanceProperties").addChild("instance"); 
prop.setAttribute("clientId",clientId);
prop.setAttribute("applicationId",applicationId);
prop.setAttribute("instanceId",instanceId);
di=req.getChild("instanceProperties").addChild("db2Instance");
// set the user name
di.addChild("userAccount").addChild("userName").setValue(instanceName);
sd = di.addChild("DB2StorageDevice");
sd.addChild("commandLineStoragePolicy").addChild("storagePolicyName").setValue(xpath:{/workflow/ExecuteProcessBlock_DB2GetDataPolicy/Script_GetPolicyName/output});
sd.addChild("dataBackupStoragePolicy").addChild("storagePolicyName").setValue(xpath:{/workflow/ExecuteProcessBlock_DB2GetDataPolicy/Script_GetPolicyName/output});
sd.addChild("logBackupStoragePolicy").addChild("storagePolicyName").setValue(xpath:{/workflow/ExecuteProcessBlock_DB2GetLogPolicy/Script_GetPolicyName/output});

return req.serialize();


TIP 8 - Error Handling from Activity Control

Not related to your original question, but many times see this misunderstood in customers workflow code, so this may be helpful to you or others

Hope the above is in someway helpful

Regards

-Roger

Userlevel 2
Badge +1

Hi @Graham Swift 

API is in some parts even worse if I’m honest:sweat_smile: Already using it to configure Things that weren’t available as QCommand-Activities or that do not have xml-Requests documented, like setting OperationWindows. It is still super limited in FeatureSet (as of FR20). Intended to use it for Automation to register and configure new Clients back in SP14, but half of the Operations are not available and for the rest you need a university degree to collect all the needed infos from two dozen concated api calls. So Workflow we went. Not much different now. Yeah we don’t use Plans yet and still wait for them to get usable :zipper_mouth:

Documentation only shows you one or two Samples with carefully selected Attributes, but rarely explains what they mean, which are mandatory or optional and what Values are available. Having different ID-System in some places was also annoying, though that seems to be mostly aligned in fr20 now.

In general I don’t like to mix so many different Attack-Vectors. CVDB, XML-Requests and Java-Code are already plenty to have issues with. Using HTTPClient Requests and adding overhead logic to check the replies would bloat Workflow Screen quite a bit. Makes more sense using them in fully custom written API-Client.

Anyhow, thanks for the hint!

Thanks @Stefan Vollrath it is great to get the feedback, and yes I agree we do have some improvements to make. As you can see in other replies here we are actively working on that side of things. One thing that may also help is in FR21 - Equivalent API - “button” lets you build out what it is you are doing in the Command Center and then let you see the JSON payload as you created it. 

It is not everywhere but may help out in some way:)

https://documentation.commvault.com/11.21/essential/128016_downloading_api_json_payload_for_command_center_operations.html

Userlevel 1
Badge +3

Hi @Stefan Vollrath, would adding more details about the attributes and more example help? What would make it easy and more usable

Hi @Seema Ghai, I would in general wish for a full list of all Methods and Functions you added for Scripting-Blocks to interact with other Workflow Activities. Creating and updating Variables is documented, how to effectively work with activity-Replies or what updates are possible there is not. How to set a workflow failed is stated, how to complete(with Error) it via Script is not. And whole Set of options available to query CSDB and how to execute xml-Requests (securely) via Script hasn’t been documented to my knowledge either. Depending on how complex the Parameter-Set is some Samples would be nice, if they all are just function(string value) one would be plenty IMHO.

Userlevel 1
Badge +3

Thanks a lot @Roger! Read through your post twice and think I still just get 15% or so :-D Is there some Change-Note that lists the differences between the old and new Java/JavaScript engines? Or a general summary of the included libraries (without having to dig through the WFE Dirs) or recommended ways to interact with xml/json data? Would use the phrase Best Practise but that seems to be one that some Vaulters don’t like ;-) Would be annoying to use one way now in FR20 that suddenly won’t work under new releases anymore.

The xml Payload creation and modification looks powerful, maybe a little scary. Compared to the xml-Scripts ingested by QOperation those don’t seem to care if something is an attribute or its own element. Is that the case or are there limitations or facts to consider? Like same attribute name used in different elements of the xml for different use-cases.

Workflow Element Classes Definition and the other Slides look interesting. Are those infos available for ESP Customers or does one have to go through workflow courses to get to these infos?

Also thanks for the Error Handling Tips, figured those out via weeks of trial and error, and even with that fore-knowledge stumble over it sometimes.

Again, thanks a lot and sorry for the continuing silly questions.

Userlevel 1
Badge +3

Thanks for the hint @Graham Swift , did see that new Feature in ChangeNotes. For once one that did make it in GA and not just BOL, or the other way around :joy: Just finished switching all Envs to FR20 hoping for more stability, so not there yet. 

Userlevel 1
Badge +1

Thanks a lot for your feedback @Stefan Vollrath. We will work on enhancing our API documentation to include more information.

 

Userlevel 1
Badge +1

Hi Stefan,

Nothing officially published regarding the Java/JS engine changes that I know of. This is just what I’ve found in the process of using it, and my own observations.

The scriptable Java within Workflow  (I think from memory) it was Beanshell 1.6 FR20 and prior,  and is now (again from memory as I’m not near a CommCell) Groovy 3.03 as of FR21 - Groovy seems to be a full superset of Beanshell’s capability (so no breaking changes that I’ve found) with lots of additional features and a far more modern set of syntactic sugar more in line with the underlying JDK Java language features, so the underlying Groovy language site itself may be a good reference.

Here is one of the few direct comparisons I could find

https://dzone.com/articles/groovy-vs-beanshell-making-the-right-decision#:~:text=BeanShell%20is%20a%20small%2C%20free,language%20features%2C%20written%20in%20Java.&text=Apache%20Groovy%2C%20on%20the%20other,language%20for%20the%20Java%20platform.

 

With Javascript, it remains the Nashorn Engine (The JS Engine within the JDK), and the JDK remains 11.0.7 and therefore the Nashorn Engine version is Oracle Nashorn 11.0.7.

Differences, As best I can tell from my observations:

  • It appears *scripting* mode in Nashorn javascript has been disabled as of FR21, this means some of the mozilla JS extensions that work in FR20 and previous won’t work in FR21 (e.g. for each, or function closures) see screenshots below.
  • However it appears the JS ecmascript version has jumped from es5.1 to some variant of es6 (suspect ecmascript 2015) – so this means some new javascript features become available for (foo in bah), or (foo of bah) etc, again see screenshots below

 

This does run the possibility of some previous JS script not running in FR21 if it uses functions exposed by the  Mozilla extensions / --scripting capabilities,  (e.g I started going down this rabbit hole after a Workflow I wrote stopped on a for each (item in array) piece of javascript. And also  the reverse, e.g. if you use ecmascript 2015 capabilities in a FR21+ workflow and then load it back in a  FR20 or prior CommCell.

 Heres some more info that talks to the nashorn extensions that look to me like they cease to be usable in FR21 https://wiki.openjdk.java.net/display/Nashorn/Nashorn+extensions

 

Finally a couple of screenshots showing some of the diagnostics that lead me to these  observations:
 

Resulting in the below output (this output has the FR21 compatible section uncommented, and the FR20 commented out - as its run on FR21)


@Chris Sunderland - If you get a chance, can you confirm or deny any of the above  if possible please?

 

The slides I referenced previously were from an Internal one-off training program written by my team at Commvault, they don’t form part of any publically facing material. 

Regards

-Roger

Reply