Each Compliance rule can be augmented with additional rule metadata through the use of Attributes and Tags, each of annotations, the most significant of which is the Tag annotation which is described here.

Rule

...

Here’s an example of a rule definition.

# Rule attributes
@AssetType("data_intg_flow")
@AssetType("data_intg_subflow")
package datamigrators

@Documentation(“http://docs.ibm.com/cp4d/compliance/Job does_not_contain_unit_test_parameter”)
@SomeAttribute(“Some value”)
@SomeOtherAttribute(“Some other value”)

# Rule tags (effectively user-defined, free-form attributes)
@Tag(“security, portability, maintainability, fail-ci, CorpDataWarehouse, CheeseSandwichTeam”)

# Rule definition
if (!(item.parameters.any { paramName, param -> paramName.contains("DM_ENABLE_UNIT_TESTING") })) {
    compliance.failure("MettleCI Unit Test job parameter has not been added to this job");
};

Rule Tags

Each MettleCI Compliance rule can include extra metadata using a set of user-specified values which are referred to as ‘tags’.

Tagging supports a number of functions:

...

Identifies which asset type(s) each rule applies to. Currently a single rule (e.g. ‘Database Connect Not Auto Generate SQL’) needs to exist as four files within the Compliance repository (‘.pjb.grm’, ‘psc.grm’,. ‘sjb.grm’, and ‘ssc.grm’) with each file effectively repeating the rule but with a different file extension to associate it with the asset to which it can be applied. Tagging will replace this mechanism by moving the name of the asset type(s) to which a file can be applied from file extension to tags defined within the rule, thereby removing the need for multiple, redundant versions of the same file to be maintained within the Compliance repository. This capability also easily enables the future application of Compliance to non-DataStage asset types.

...

Annotations

Annotations add extra information to a Compliance rule for use by the MettleCI Workbench and CLI to change their behaviour. Annotations consist of a keyword starting with an @ character followed by relevant metadata. The only annotation you need to use on a day-to-day basis is the @Tag annotation which, although optional, is strongly recommended.

Rule Tags

Each Compliance rule may specify zero or more tags which are free-form text labels associated with each rule. You may choose any number of tags, and each tag may take any alphanumeric value you choose. Rule Tagging provides improved rule metadata management in support of a number of use cases:

• Identifying the ‘severity’ of each rule. The MettleCI tools that use Compliance results (the Workbench user interface and the mettleci compliance test CLI command) need to know how to respond to Compliance Rule breaches.

...

• How a rule’s breach should

...

• be indicated in Workbench or

...

• whether its use in a CI pipeline should produce a warning message or abort the pipeline entirely can be defined using Tags.

• Grouping Compliance Rules into ‘bundles’ of functionally related rules. This could enable users to report or test by functional area. Functional groups into which Compliance Rules could be bundled might be Performance, Security, Maintainability, etc. Tagging also permits a single rule to be associated with multiple functional areas, if required.

Its features are:

• Each rule specifies zero or more free form ‘tags’

• Rule Tagging improves metadata management

  Fine

  Enabling the fine-grained sharing of rules across teams within organisations (i.e. tags could be used to identify which DataStage

  projects

  teams they apply to)

• Define

  Defining rules’ behaviour in different environments (e.g. Workbench vs. CI/CD plans)

The

...

This has been retrofitted to MettleCI for standalone DataStage

• The Workbench permits the selection of rules using positive and negative rule tag values

• Legacy CLI also updated to support tags

• This sets the expectation that this capability is also available in Cloud Pak!

...

example CI/CD build pipelines that ship with MettleCI demonstrate the use of tags to identify which rules inhibit the successful completion of CI.

Some notes on Tag behaviour:

...

Tags are case insensitive.

...

The default behaviour of not specifying any ‘Include’ tags is that everything is included.

...

The default behaviour of not specifying any ‘Exclude’ tags is that nothing is excluded.

...

When no ‘Include’ and ‘Exclude’ tags are specified Compliance Rules are not scanned recursively (existing behaviour).  When at least one tag is specified (either Include or Exclude), rules are scanned recursively.

...

Using Tags

Here’s an example of a rule definition which incorporates some Tags.

# Rule tags (effectively user-defined, free-form attributes)
@Tag("security")            # This rule identifies a potential security vulnerability
@Tag("portability")         # This rule identifies a issues with assets' portability between environments
@Tag("maintainability")     # This rule identifies a potential maintainability issue
@Tag("CorpDataWarehouse")   # This rule is specific to the 'CorpDataWarehouse' team
@Tag("fail-ci")             # This rule is mandatory and so should fail continuous integration if breached

# Rule attributes
@AssetType("data_intg_flow")
package datamigrators

# Rule definition
<blah blah blah>

Include and Exclude options for Compliance operations

The various MettleCI tools which use the Compliance Rule library permit the filtering of the rules which they use by allowing users to specify which tags should cause a rule to be included and which excluded.

...

The mettleci compliance test command (available within the Compliance namespace of the MettleCI Command Line Interface) provides -include-tagtags and -exclude-tagtags options to filter the rules which will are used in your call to the compliance test command. The way the tags are considered means be used by the command. MettleCI interprets these options so that set of include tags is used before first to generate a set of candidate rules, then the set of negative exclude tags is removed to leave the rules that will be used. In the diagram above, only the rules with tags in set A and NOT in B will be used to select Compliance Rules (formally referred to as the 'relative complement of B in A').

Tags Example

$> mettleci compliance test \
   -assets /my/datastage/assets \
   -report "my/compliance/report.xml" \
   -junit \
   -rules /my/compliance/directory \
   -exclude-tags example \
   -project-cache /my/project/cache \
   -ignore-test-failures \
   -test-suite "CI Warnings"

MettleCI Command Line (build 161)
(C) 2018-2022 Data Migrators Pty Ltd
compliance test (v2.2-425)
rules configuration discovered
included rule - 'Adjacent Transformers' (PARALLEL_JOB)
... etc.

Some notes on Tag behaviour:

• Tags are case insensitive.

• We strongly recommend that the values your use for Tags employ only alphanumeric characters (0-9, a-Z, A-Z) as Data Migrators can’t guarantee the support for non-alphanumeric tags across all possible CI platforms and use cases.

• The default behaviour of tagging is that everything is included:

  • Omitting an Include tag option results in everything being included.

  • Omitting an Exclude tag option results in nothing being excluded (i.e. everything is included).

  • Omitting Include and Exclude tag options results in everything being included.

• When at least one tag is specified (either Include or Exclude), Compliance Rules are scanned recursively.

• When no Include and Exclude tags are specified Compliance Rules are not scanned recursively (existing behaviour). 

• You can use the asterisk (*) character to match all tags, which can be a useful technique to trigger recursive scanning when you need it:

  • Specify an Include tag using * (to include everything) and leave Exclude tags blank to trigger recursive scanning of rules. This is different to omitting both specifications (or leaving them blank) which would results in shallow, non-recursive scanning.

• Multiple tags are expressed as a comma separated list with no spaces. (e.g. -include-tags MyTag,AnotherTag,YetAnotherTag)

Workbench Configuration

The MettleCI Workbench permits you to specify which include and exclude tags will be used when testing your Job’s Compliance interactively from within the Workbench interface.

Image Added

Image Added