Some of the sample build pipelines that ship with MettleCI specify the -ignore-test-failures option when testing for both warning and fatal compliance rules. Intuition may lead you to think that we shouldn’t ignore test failures for Compliance Rules tagged as fatal, but this depends on the build system you’re using.
Here’s the pseudo-code of the mettleci compliance test command (documentation) as used in thecompliance_template.yml file used in the Azure pipeline example to invoked invoke compliance testing for both warning rules and fatal rules:
| Code Block |
|---|
mettleci compliance test
-assets {location}
-report {reportname}
-junit
-rules {compliance_repository}
-include {include_tags}
-exclude {exclude_tags}
-project-cache {location}
-include-job-in-test-name
-test-suite {test_suite_name}
-ignore-test-failures |
The -ignore-test-failures option determines whether any test failure will cause the mettleci compliance test command (docs) to generate a non-zero return code to the shell from which it was invoked. Some build systems, like Azure DevOps, for example, will respond to interpret any non-zero return code as a pipeline failure (which is close to what we want) but will also fail to process any generated and will consequently cease pipeline execution and ignore the contents of any JUnit XML test reports , like those generated as a by-product of the failing command. In this case, use of the -ignore-test-failures option in an Azure pipeline to allow allows the pipeline to continue past the point of the test execution , and then configure the step processing the process the resulting JUnit XML test results to class files as test failures. The treatment of these test failures as either Warnings (noteworthy, but acceptable to continue processing), or Errors (critical job failures: abort the pipeline)or Fatal will then dependent upon your use of the ContinueOnError (documentation) setting in your pipeline definition.