Here is a summary of the Upgrade pipeline along with a detailed description of each Stage, Job, and Task. In the example below Test_Legacy_CI is a Stage, Compliance_Test_WARN is a Job within that Stage, and the tasks within this Job are detailed in the page below. Different build tools employ different terminology, but Stage, Job and Task are commonly adopted.

This example uses a screenshot from Microsoft Azure DevOps (below), but the process is identical regardless of the build tool being used. You can use this description to create a MettleCI build pipeline using a build tool for which we do not currently provide a. pipeline template.

Select the pipeline stage to jump to its description:

Notes

Note that every build Job has implicit access to the DataStage asset repository where this pipeline definition is stored. Some build tools make this explicit and produce a log entry showing this. For example Azure DevOps automatically introduces a task at the beginning of each Job with a title for the form Checkout <repository-name@<branch> to s\<project-name>.
Some of the pipeline templates (e.g. deploy_template.yml) are parameterised, meaning they may perform different tasks based on the context within which they are invoked (e.g. deploying to legacy vs. deploying to target)

Upgrade Pipeline - Stages, Jobs, and Tasks

1. Diagnostics

This stage is simply a diagnostic step to help with initial development of your pipeline. Once you have confirmed that your pipeline is behaving as required you can safely delete this step.

Diagnostics

#	Task	Command	Description
1	Diagnostics v11.7	various	Uses the Build Agent to log environment variables and variable groups for diagnostics purposes.

2. Create DSParams Diff Artifact

Create DSParams Diff Artifact

#	Task	Command	Description
1	Fetch template DSParams	`mettleci remote download` (documentation)	Fetch the template `DSParams` file from the legacy DataStage engine.
2	Create artefacts directory	Inline script	Verify that the build directory `artefacts`exists (otherwise, create it).
3	Generate diff	`mettle dsparams diff` (documentation)	Generate a `DSParams.diff` file of differences between template `DSParams` and actual `DSParams`.
4	Publish diff artefact	Build tool specific	Define the `DSParams.diff` file as an artefact to make it available to other pipeline stages.

3. Deploy CI

Deploy to Continuous Integration

This Job is defined in pipeline template deploy_template.yml

#	Task	Command	Description
1	Download artefact	Build tool specific	Download the previously-created `DSParams.diff` file to the agent’s local directory.
2	Create project	`mettleci datastage create-project` (documentation)	Verify that the CI project exists (otherwise, create it).
3	Configure properties	`mettleci properties config` (documentation)	Replace variables in the `DSParams` file, any Parameter Sets files, and any user-define shell scripts (from repository path `/datastage`) with values defined in the `var.ci` properties file. Processed files are stored in a `config` directory created by this command in the current workspace on the Azure agent.
4	Upload config files	`mettleci remote upload` (documentation)	Upload the `config` directory containing files modified by the Configure Properties step and your repository’s `filesystem` directory from the build agent to your CI environment.
5	Execute deploy script	`mettleci remote execute` (documentation)	Execute your `config/deploy.sh` script (previously generated by the Configure properties task from your repository's `datastage/deploy.sh` file) on your CI environment. This user-supplied script performs solution-specific deployment processes required by the assets in the `filesystem` directory.
6	Deploy DataStage assets	`mettleci datastage deploy` (documentation)	Perform an incremental deployment of your DataStage assets to the CI project.
7	Cleanup	`mettleci remote execute` (documentation)	Execute the script `config/cleanup.sh` (previously generated by the Configure properties task from your repository's `datastage/cleanup.sh` file) in the CI environment. This script removes files which are no longer required from your build agent’s disk.
8	Check for results	Inline script	Detects and records (in a build pipeline variable) whether the Job compilation performed by the Deploy DataStage assets task produced an output jUnit `.xml` file. Note that the incremental deployment approach used by MettleCI may mean that a test output file would not be generated in the event that the deployment process does not detect any changes.
9	Publish compilation results	Build tool specific	If a jUnit `.xml` file was detected by the Check for results step then publish it to the build tool for use in test suite management.

4. Test CI

Compliance Test Warnings

This Job is defined in pipeline template compliance_template.yml.

#	Task	Command	Description
1	Checkout Compliance repository	Build tool specific	Make the Compliance repository (separate to the DataStage asset repository, where this pipeline definition is stored) accessible to the build agent.
2	Run Compliance Warnings	`mettleci compliance test -ignore-test-failures` (documentation)	Invokes MettleCI Compliance checks for changes jobs. Note that in this mode (including the `-ignore-test-failures` option) a failed Compliance check will not cause the entire build pipeline to fail.
3	Check for results	Build tool specific	Detects and records (in a build pipeline variable) whether the Compliance check performed by the Run Compliance task produced an output jUnit `.xml` file. Note that the incremental deployment approach used by MettleCI may mean that a test output file would not be generated in the event that the deployment process does not detect any changes.
4	Publish compliance results	Build tool specific	If a jUnit `.xml` file was detected by the Check for results step then publish it to the build tool for use in test suite management.

Compliance Test Failures

This Job is defined in pipeline template compliance_template.yml

#	Task	Command	Description
1	Checkout Compliance repository	Build tool specific	Make the Compliance repository (separate to the DataStage asset repository, where this pipeline definition is stored) accessible to the build agent.
2	Run Compliance (CI FAIL)	`mettleci compliance test` (documentation)	Invoke MettleCI Compliance checks for changes jobs. Note that in this mode (omitting the `-ignore-test-failures` option) a failed Compliance check will cause the entire build pipeline to fail.
3	Check for results	Inline script	Detects and records (in a build pipeline variable) whether the Compliance check performed by the Run Compliance task produced an output jUnit `.xml` file. Note that the incremental deployment approach used by MettleCI may mean that a test output file would not be generated in the event that the deployment process does not detect any changes.
4	Publish compliance results	Build tool specific	If a jUnit `.xml` file was detected by the Check for results step then publish it to the build tool for use in test suite management.

Unit Test

This Job is defined in pipeline template unittest_template.yml

#	Task	Command	Description
1	Upload config files	`mettleci remote upload` (documentation)	Upload the file `datastage/cleanup_unittest.sh` from the build agent to your CI environment.
2	Upload unit test specs	`mettleci remote upload` (documentation)	Upload your repository’s `unittest` directory from the build agent to your CI environment.
3	Configure properties	`mettleci properties config` (documentation)	Replace variables in the `datastage/cleanup_unittest.sh` file with values defined in the `var.ci` properties file and store the processed file in the `config` directory.
4	Create unit test report dir	`mkdir unittest-reports`	Create directory to receive unit test outputs.
5	Run Unit Tests	`mettleci unittest test` (documentation)	Invoke the Unit Tests defined in your `unittest` directory. Note that MettleCI will use its incremental detection functionality to only execute Unit Tests where either the Unit Test case or associated Job have changed.
6	Download unit test reports	`mettleci remote download` (documentation)	Download any jUnit `.xml` files produced by the Run Unit Tests task from your CI environment to the build agent.
7	Cleanup	`mettleci remote execute` (documentation)	Execute the script stored in the repository file `config/cleanup_unittest.sh` (previously generated by the Configure properties task) in the CI environment. This script removes unit test results, to ensure a clean environment for the next execution.
8	Check for results	Inline script	Detects and records (in a build pipeline variable) whether the Unit Test execution performed by the Run Unit Tests task produced an output jUnit `.xml` file. Note that the incremental deployment approach used by MettleCI may mean that a Unit Test output file would not be generated in the event that the deployment process does not detect any changes.
9	Publish compilation results	Build tool specific	If a jUnit `.xml` file was detected by the Check for results step then publish it to the build tool for use in test suite management

5. Deploy Target CI

Deploy to Upgrade Target - Continuous Integration