Structure

The Jenkins DevOps pipeline is defined in a “Jenkinsfile” (by default named Jenkinsfile ) and has the following structure (most detail omitted). Paired curly braces{ } denote the start and end of a scoped section.

pipeline {
    agent { ... }
    parameters { ... }
    environment { ... }
    stages {
          stage { ... }
          stage { ... }
          ...
    }
}

To understand these sections in more detail, refer to your Jenkins documentation. We present only the MettleCI specific things you need to know. Note that Jenkins offers flexibility in where some of these sections are placed, and in some cases, allows repetition at different scopes. (The {} are scope delimiters.) Note also that there are additional things you could put in to just about every section, and that some sections may be repeated at a given scope level.

Sections before “stages”

Agent section

The Agent agent section denotes what class of machines your pipeline can be executed on. For a CI/CD pipeline, you only need to specify this once since all projects are at the same release level. For an upgrade pipeline you may want to specify the agents within the stage so that different stages run on different release levels. In the example below the label should match the label you gave your agent on the MettleCI host when you set it up. Pipelines will stall if no running agent with the (matching set of) label(s) is found.

    agent {
        label 'mettleci:datastage11.7.1'
    }

Parameters section

The Parameters parameters section enumerates the parameters to the pipeline with their default values. The four parameters used in the sample, and their meanings are discussed in Deploying a Jenkins Pipeline …

    parameters {
        string(name: 'domainName', defaultValue: 'demo4-svcs.datamigrators.io:59445', description: 'DataStage Service Tier')
        string(name: 'serverName', defaultValue: 'demo4-engn.datamigrators.io', description: 'DataStage Engine Tier')
        string(name: 'projectName', defaultValue: 'wwi_jenkins_ds117', description: 'Logical (unsuffixed) Project Name')
        string(name: 'environmentId', defaultValue: 'ci', description: 'Environment Identifer')
    }

When run interactively, the user will be prompted for these at pipeline start time like so:

...

...

Environment Section

The environment section allows you to derive or and set further useful variables. The sample pipeline composes the suffixed DataStage project and copies one of the parameters to an all uppercased variable for convenience later.

    environment {
        DATASTAGE_PROJECT = "${params.projectName}_${params.environmentId}"
        ENVIRONMENT_ID = "${params.environmentId}"
    }

These will be visible in the execution environment as environment variables, and thus will be available to batch files, external commands, and the like without needing to explicitly pass them.

Stages Section

The stages section is a container for an arbitrary number of individual stages, which are normally run in sequence. In the sample pipeline there are three, “Deploy”, “Test”, and “Promote”

Stage Section notes

Each stage section begins with a label, which is a quoted arbitrary text string displayed in Blue Ocean when viewing the pipeline,

Within the stage are two major subsections, steps{} and post{}, which contain the actions to be performed during normal execution, and the actions to be performed at the end of the stage

Use of the parallel { } construct allows nesting of stages within stages.

Deploy Stage

This is the first stage in the pipeline and it is tasked with deploying to the CI (continuous integration) project. WithCredentials provides appropriate credentials (using the credentials plugin) to the rest of the steps. change from mci-user if you are using a different

 stage("Deploy") {

            steps {
                withCredentials([usernamePassword(credentialsId: 'mci-user', passwordVariable: 'datastagePassword', usernameVariable: 'datastageUsername')]) {

                    bat label: 'Create DataStage Project', script: "${env.METTLE_SHELL} datastage create-project -domain ${params.domainName} -server ${params.serverName} -project ${env.DATASTAGE_PROJECT} -username ${datastageUsername} -password ${datastagePassword}"
                    
					bat label: 'Substitute parameters in DataStage config', script: "${env.METTLE_SHELL} properties config -baseDir datastage -filePattern \"*.sh\" -filePattern \"DSParams\" -filePattern \"Parameter Sets/*/*\" -filePattern \"*.apt\" -properties var.${params.environmentId} -outDir config"

                    bat label: 'Transfer DataStage config and filesystem assets', script: "${env.METTLE_SHELL} remote upload -host ${params.serverName} -username ${datastageUsername} -password ${datastagePassword} -transferPattern \"filesystem/**/*,config/*\" -destination \"${env.DATASTAGE_PROJECT}\""

                    bat label: 'Deploy DataStage config and file system assets', script: "${env.METTLE_SHELL} remote execute -host ${params.serverName} -username ${datastageUsername} -password ${datastagePassword} -script \"config\\deploy.sh\""
                    
                    bat label: 'Deploy DataStage project', script: "${env.METTLE_SHELL} datastage deploy -domain ${params.domainName} -server ${params.serverName} -project ${env.DATASTAGE_PROJECT} -username ${datastageUsername} -password ${datastagePassword} -assets datastage -parameter-sets \"config\\Parameter Sets\" -threads 8 -project-cache \"C:\\dm\\mci\\cache\\${params.serverName}\\${env.DATASTAGE_PROJECT}\""

                }
            }
            post {
                always {
                    junit testResults: 'log/**/mettleci_compilation.xml', allowEmptyResults: true
 /*                   withCredentials([usernamePassword(credentialsId: 'mci-user', passwordVariable: 'datastagePassword', usernameVariable: 'datastageUsername')]) {
                        bat label: 'Cleanup temporary files', script: "${env.METTLE_SHELL} remote execute -host ${params.serverName} -username ${datastageUsername} -password ${datastagePassword} -script \"config\\cleanup.sh\""
                    }
                    deleteDir()  */
                }
            } 
        }