| Table of Contents |
|---|
Introduction
The development environment consists of a simple command line interface (Java-based, with scripts for both Windows and Linux provided) that loads the MettleCI Compliance plugin and allows Compliance Rules to be executed against the exported DataStage jobs.The MettleCI suite as installed features a number of standard Compliance Rules that it executes against the ISX archive files it creates to manage DataStage jobs.
...
NOTE: As of January 2019, this development capability is separate to the MettleCI installation and rules created by end users need to be managed and promoted to MettleCI manually.
Setting up a Local Development Environment
Note: Even though the The steps involved in setup of the Command Shell tool in both Windows or Linux is the same.
To execute Compliance Rules against ISX files, 1)
Copy the DM Command Shell (with a name similar to
dm-mettleci-command-shell-n.n-nnn-dist.zip) from your MettleCI installation media (See Accessing the MettleCI Software Distribution) and unzip to a convenient directory.
...
The Directory Structure that is unzipped should look like this:
...
Edit
config.properties, and on the line forlicense.file, Enter the name of the MettleCI license file, relative to this directory.
...
You should have a
...
copy of the Compliance Plugin (
dm-compliance-plugin-2.0-*.jar) into thepluginsdirectory, if not search for thedm-compliance-plugin-2.0-*.jarin the MettleCI installation media and copy it into thepluginsdirectory shown above.
...
The directory lib stores a few dependencies for the compliance command line execution, the directory should have the following items:
...
commons-io-2.4.jar
...
commons-lang-2.6.jar
...
...
Open a Command Prompt, and Run 'shell' and type 'help'. The shell returns Namespace 'compliance' which shows that the Compliance plugin has been copied into the plugins directory and is available for use.
...
7) Execute 'compliance help' to see the required parameters for executing a compliance test.
...
| Note |
|---|
In order for a file to be recognised as MettleCI Compliance Rules, it must have the file extension " |
9) Sample output from the Compliance Test:-
...
10) In order to speed up execution, JSON files are not saved by default.
...
Note that the filter parameter was set to "." to ensure the entire json document was processed.
Compliance Rule Development Concepts
Rules Relate to Specific DataStage Asset Types
...
Refers to the entire graph, under which two main collections exist:
V - Vertices - all Vertices in the model: Stages, Pins, Metabags, Containers, Annotations
E - Edges - all Edges in the model: Links and Pins (the Edges that connects a Stage to a Pin)
...
Vertices are connected by Edges, and the graph can be walked Vertex by Vertex using:
.in - Inbound (Upstream) Vertices: Refers to the Vertex at the end of an inbound Edge
.out - Outbound (Downstream) Vertices: Refers to the Vertex at the end of an outbound Edge
.inE - Inbound Edges connecting to upstream Vertices: Refers to the outbound Edges.
.outE - Outbound Edges connecting to downstream Vertices: Refers to the inbound Edges.
V and .in and .out are used in conjunction with a Vertex filter. MettleCI creates three of these filters to allow for more concise coding of rules.
.stage- filter the Vertices to just the Stages and is a shortcut for .has('type', 'stage')
.pin - either end of a Link
.metabag - for a Stage, it is the contents of the Advanced tab. For a Pin it is the contents of the Input or Output tab of the adjacent Stage (depending on how it was traversed to).
If traversal involves Edges:
.inE - Inbound Edges connecting to upstream Vertices: Refers to the outbound Edges. Edges have a label property of either "link" or "pin"
.outE - Outbound Edges connecting to downstream Vertices: Refers to the inbound Edges.
...
E and .inE and .outE are used in conjunction with an Edge filter. MettleCI creates two of these:
.link - filter the Edges to those that are the actual DataStage Link (connects directly to the adjacent Stage) and is a shortcut for .has('label', 'link')
.pin - filter the Edges to those that connect to a Pin (which connects to the Metabag containing Stage Input or Output tab properties)
...
In this example:
def B = item.graph.V.stage.has("stageName", "txRICheck")- B refers the the txRICheck stageB.in- refers to A, and the Pin containing the Metabag that stores the properties of the Input tab in the txRICheck Stage PropertiesB.in.stage- refers to A (luRI)B.in.pin.out.metabag- refers to the Metabag containing the properties of the input tab. Note there is only one input link.B.out.stages- refers to C (scDisRIAccounts) and D (txAudit)B.out.pin.as("pin").outE.has('linkName', 'RIChecked').back("pin").out.metabag- refers to the Metabag containing the properties of the output tab for the Link named "RIChecked"
There are two primary ways of filtering to find the objects that meet certain criteria. They are used most commonly on Vertices, as Edges have few properties of interest in MettleCI.
.has(<property>,<value>) - if the Vertex contains a property and the value of that property matches the value specified, the object is retained for evaluation.
.filter{...} - behaves similarly to an function call, and returns true or false. If true is returned, the object is retained for evaluation. This allows for more complex logic to be used. The function code may be defined prior and used like a function pointer, or defined inline.
The Comply Pipe
.comply{<message closure>}, {<compliace closure>} - Comply Pipe
The first parameter is closure which returns the compliance message that gets output for each failure.
...
So the expression 'item.graph.V.stage.comply{message closure}{comply closure}' means 'In the Vertices graph, for all the items of stages, check if they comply with {comply closure}. If false, output {Message Closure}.
The Debug Pipe
.debug(<output file name>) - Debug Pipe
While the 'debug' rule described above is great for dumping the entire graph in JSON format, there are times that you will want to debug specific graph Vertices and Edges. There are also some properties, such as sort keys and connector properties, which cannot be described in a valid JSON format.
...
In the StageTypes Properties in DataStage, we can see the same options, like "RejectFromLink" on Line 11.
...
So it is possible to use any of these Attributes in Compliance Rules.
Some stages properties such as Sort Key Ascending/Descending or Connector Properties are hierarchical in nature and cannot be easily understood by looking at Stage Type properties. In these cases, use the .debug() pipe describe above to inspect all properties of any Vertex (ie. Stage, pin, metabag, etc) or Edge (ie. Links, etc)
A 'Graph Modelling Language' (GML) notation version of the graph can also be output as a JSON file, which allows one to check the various attributes of the Stages and Links, usually for debugging purposes.
...