Some of the sample build pipelines that ship with MettleCI specify the -ignore-test-failures option when testing for both warning and fatal compliance rules. Intuition may lead you to think that we shouldn’t ignore test failures for Compliance Rules tagged as fatal, but this depends on the build system you’re using.
Here’s the pseudo-code of the compliance_template.yml file used in the Azure pipeline example to invoked compliance testing for both warning rules and fatal rules:
mettleci compliance test
-assets {location}
-report {reportname}
-junit
-rules {compliance_repository}
-include {tags}
-exclude {tags}
-project-cache {location}
-include-job-in-test-name
-test-suite {test_suite_name}
-ignore-test-failures
The -ignore-test-failures option determines whether any test failure will cause the mettleci compliance test command (docs) to generate a non-zero return code to the shell from which it was invoked. Some build systems, like Azure DevOps, for example, will respond to any non-zero return code as a pipeline failure (which is close to what we want) but will also fail to process any generated JUnit XML test reports, like those generated by the command. In this case, use the -ignore-test-failures option in an Azure pipeline to allow the pipeline to continue past the point of the test execution, and then configure the step processing the JUnit XML test results to class test failures as Warnings (noteworthy, but acceptable to continue processing), or Errors (critical job failures: abort the pipeline)