Some of the sample build pipelines that ship with MettleCI specify the -ignore-test-failures option when testing for both warning and fatal compliance rules. Intuition may lead you to think that we shouldn’t ignore test failures for Compliance Rules tagged as fatal, but this depends on the build system you’re using.

Here’s the pseudo-code of the mettleci compliance test command (documentation) as used in thecompliance_template.yml file used in the Azure pipeline example to invoke compliance testing for both warning rules and fatal rules:

mettleci compliance test
  -assets {location}
  -report {reportname}
  -junit
  -rules {compliance_repository}
  -include {include_tags}
  -exclude {exclude_tags}
  -project-cache {location}
  -include-job-in-test-name
  -test-suite {test_suite_name}
  -ignore-test-failures

The -ignore-test-failures option determines whether any test failure will cause the compliance test command to generate a non-zero return code to the shell from which it was invoked. Some build systems, like Azure DevOps, for example, will interpret any non-zero return code as a pipeline failure and will consequently cease pipeline execution and ignore the contents of any JUnit XML test reports generated as a by-product of the failing command. In this case, use of the -ignore-test-failures option in an Azure pipeline allows the pipeline to continue past the point of the test execution and process the resulting JUnit XML files as test failures. The treatment of these test failures as either Warnings or Fatal will then dependent upon your use of the ContinueOnError (documentation) setting in your pipeline definition.

MettleCI Documentation

Why should some pipelines use 'ignore-test-failure' when testing fatal Compliance rules?

See also