Compliance Rule Metadata

Each Compliance rule can be augmented with additional rule metadata through the use of Attributes and Tags, each of which is described here.

Rule Attributes

Here’s an example of a rule definition.

# Rule attributes
@AssetType("data_intg_flow")
@AssetType("data_intg_subflow")
package datamigrators

@Documentation(“http://docs.ibm.com/cp4d/compliance/Job does_not_contain_unit_test_parameter”)
@SomeAttribute(“Some value”)
@SomeOtherAttribute(“Some other value”)

# Rule tags (effectively user-defined, free-form attributes)
@Tag(“security, portability, maintainability, fail-ci, CorpDataWarehouse, CheeseSandwichTeam”)

# Rule definition
if (!(item.parameters.any { paramName, param -> paramName.contains("DM_ENABLE_UNIT_TESTING") })) {
    compliance.failure("MettleCI Unit Test job parameter has not been added to this job");
};

Rule Tags

Each MettleCI Compliance rule can include extra metadata using a set of user-specified values which are referred to as ‘tags’.

Tagging supports a number of functions:

1. Identifies which asset type(s) each rule applies to. Currently a single rule (e.g. ‘Database Connect Not Auto Generate SQL’) needs to exist as four files within the Compliance repository (‘.pjb.grm’, ‘psc.grm’,. ‘sjb.grm’, and ‘ssc.grm’) with each file effectively repeating the rule but with a different file extension to associate it with the asset to which it can be applied. Tagging will replace this mechanism by moving the name of the asset type(s) to which a file can be applied from file extension to tags defined within the rule, thereby removing the need for multiple, redundant versions of the same file to be maintained within the Compliance repository. This capability also easily enables the future application of Compliance to non-DataStage asset types.

2. Identifies the ‘severity’ of each rule. The MettleCI tools that use Compliance results (the Workbench user interface and the mettleci compliance test CLI command) need to know how to respond to Compliance Rule breaches. Whether a rule’s breach should produce an abort (Fail) or informational (Warning) in Workbench or CI pipeline is currently defined (somewhat awkwardly) by the folder within which the rule exists in the repository. For NextGen this metadata would be better defined using tagging.

3. Groups Compliance Rules into ‘bundles’ of functionally related rules. This could enable users to report or test by functional area. Functional groups into which Compliance Rules could be bundled might be Performance, Security, Maintainability, etc. Tagging also permits a single rule to be associated with multiple functional areas, if required.

Its features are:

• Each rule specifies zero or more free form ‘tags’

• Rule Tagging improves metadata management

  • Fine-grained sharing of rules across teams within organisations (i.e. tags could be used to identify which DataStage projects they apply to)

  • Define rules’ behaviour in different environments (e.g. Workbench vs. CI/CD plans)

• The CLI (for use in build and deployment pipelines) also permits the selection of rules using positive and negative rule tag values

• This has been retrofitted to MettleCI for standalone DataStage

  • The Workbench permits the selection of rules using positive and negative rule tag values

  • Legacy CLI also updated to support tags

  • This sets the expectation that this capability is also available in Cloud Pak!

• Updated pipeline examples demonstrate the use of tags to identify which rules inhibit the successful completion of CI.

Some notes on Tag behaviour:

• Tags are case insensitive.

• The default behaviour of not specifying any ‘Include’ tags is that everything is included.

• The default behaviour of not specifying any ‘Exclude’ tags is that nothing is excluded.

• When no ‘Include’ and ‘Exclude’ tags are specified Compliance Rules are not scanned recursively (existing behaviour).  When at least one tag is specified (either Include or Exclude), rules are scanned recursively.

• You can use * to match all tags so you can use * to include everything and leave Exclude tags blank. This will trigger recursive scanning of rules (different from leaving both blank which results in the existing, shallow, non-recursive behaviour).

You can use the -include-tag and -exclude-tag options to filter the rules which will are used in your call to the compliance test command. The way the tags are considered means that set of include tags is used before the set of negative tags is removed. In the diagram above, only the rules with tags in set A and NOT in B will be used to select Compliance Rules. In set theory this is referred to as the relative complement of B in A, or more formally B ∖ A.