Structure

The Azure DevOps pipeline is defined in the file azure_pipline.yml file and has the following structure:

# Preamble, setting up parameters for the pipeline
# ------------------------------------------------

trigger:                                  # The repository event(s) which trigger this pipeline
pool:                                     # Describing the Azure DevOps agents which will execute this pipeline
variables:                                # The configurable parametesrs controlling the behaviour of this pipeline

# Stages, where the actual execution steps of the pipeline are defined
# --------------------------------------------------------------------

stages:                                   # A list of stage definitions which represent the distinct logical steps in the pipeline 
- stage: Deploy_CI                        #
- stage: Deploy_QA                        # 
- stage: Deploy_Perf                      # 
- stage: Deploy_Prod                      #

The Preamble

The start of the pipeline defines global information about the pipeline itself:

# You can add more steps to your pipeline that build, run tests, deploy, and more.
# See https://aka.ms/yaml for more details

trigger:                                    # This pipeline is trigger by a commit to the associated repository's master branch
- master                                    #

pool:                                       # The pool of agents from which an agent should be assigned for execution of this pipeline
  AWS DataStage Agents                      #

variables:                                  # A set of key:value pairs used throughout the pipeline definition
  IISUsername: isadmin                      # 
  IISPassword: your-password-here           #
  MCIUsername: mciworkbench                 #
  MCIPassword: your-password-here           #
  AzureUsername: your.name@domain.com       #
  AzurePassword: your-password-here         #
  ProjectName: your-datastage-repo          #
  MettleHome: /opt/dm/mci                   #
  ComplianceRepo: your-compiance-repo       #

Stages

Each Stage defines a number of jobs and is structured like this:

- stage: StageName                        # A label for the stage
  variables:                              # The configurable parameters controlling the behaviour of this stage
  jobs:                                   # A list of job definitions (not to be confused with DataStage Jobs) which represent the distinct operational steps in the pipeline
  - job: Compliance_Test_Warning          # 
  - job: Compliance_Test_Fatal            # 
  - job: Unit_Test                        # 
  - job: Some_Other_Job                   #

Jobs

Note that an Azure DevOps Pipeline ‘Job’ is not relate to a DataStage ‘Job’. An Azure DevOps Pipeline Job is a unit of work within an Azure DevOps pipeline.

Jobs can be thought of as simple sequential scripts which perform units of work. Here’s an example job from the default azure_pipline.yml which ships with MettleCI:

Sometimes you’ll find that you need to re-use the same logic in multiple jobs across different stages. Deploying your jobs to a target environment is a good example, as your changes will need to be deployed identically to all your downstream CI, QA, and (potentially) Production environments. The best way to do this is to use a pipeline template, which is a set of jobs defined in an external file which can be called repeatedly from multiple stages. A template has a name and input parameters, and is used like this:

The Default MettleCI Pipeline

Here’s a summary of the default azure_pipline.yml which ships with MettleCI:

Also See

Azure DevOps CI Triggers

MettleCI Documentation

Understanding your Azure DevOps CI/CD Pipeline