The use of Bamboo Task-based pipelines has been superseded by the use of Bamboo-Specs based pipelines, example of which are shipped with MettleCI v1.3 onwards.

Introduction

MettleCI includes a set of standard build and deployment plans for use with Bamboo which provide a CI/CD pipeline for your DataStage projects. The MettleCI CLI includes a command which will deploy Bamboo build and deployment plans for a DataStage project. An example representation of a deployed CI/CD pipeline configured to have 3 Bamboo Deployment Environments is shown below:

This page documents the prerequisites, deployment and post-deployment steps required to get up and running with MettleCI Bamboo based CI/CD pipelines.

Prerequisites

Before attempting to deploy MettleCI Bamboo build and deployment plans, you will need to configure the following in Bamboo:

Add a DataStage Capability for your DataStage version to at least 1 Bamboo Agent.
Create linked source code repositories (all plans) for each of the following Git repositories which match (including branch name) those used to register your project with Workbench:
1. The DataStage Asset Repository for which you you will generate Bamboo Plans
2. Compliance Repository. A single Linked Repository for Compliance can be used with multiple plans so there’s no need to create multiple Linked Repositories for this if one already exists.
Create shared username and password credentials which will be used for the following
1. Logging into DataStage as a Suite Administrator using the DataStage client tools.
2. Connecting to the DataStage Engine Host using SSH with Permissions to read/write files used by your ETL project and to modify DSParams which is typically found under /opt/IBM/InformationServer/Server/Projects/<My Project Name>/DSParams

To aid in maintenance, it is recommended that a single set of credentials for each DataStage host that MettleCI will deploy to. The credentials are secured and used as a service account which MettleCI will use to deploy DataStage and File System assets during the execution of your CI/CD pipeline.

An installation of MettleCI Command Line Interface
A copy of dm-command-bamboo plugin.
You will also need to ensure that the correct versions of the Atlassian Bamboo Plan Specs Implementation and API .jar files are added to the libs directory of your MettleCI CLI home folder. You can find these files at the following URLs:

Bamboo Plan Specs Implementation https://mvnrepository.com/artifact/com.atlassian.bamboo/bamboo-specs
Bamboo Plan Specs API https://mvnrepository.com/artifact/com.atlassian.bamboo/bamboo-specs-api

On each page, click on the link that corresponds to the version of Bamboo you are running. On the resulting page, click on the “.jar” link to trigger the download.

If you have trouble setting up MettleCI CommandLine with dm-command-bamboo-plugin.

Refer to troubleshooting pages below

Deployment

The following MettleCI Command Line Interface syntax can be used to generate the Bamboo based CI/CD pipeline:

$> mettleci deploy devops-pipeline help
The following options are required: -datastage-install -plan-name -username -source-repository -plan-key
 -bamboo-shared-credentials -bamboo -password
Usage: devops-pipeline [options]
  Options:
  * -bamboo
       URL for Bamboo server
  * -username
       Bamboo username
  * -password
       Bamboo password
  * -plan-key
       Bamboo Plan key
  * -plan-name
       Bamboo Plan name
  * -source-repository
       Bamboo Linked Repository containing Project Source code
  * -datastage-install
       DataStage capability used by plan
  * -bamboo-shared-credentials
       Name of Bamboo shared credentials used when connecting to DataStage
    -project-key
       Bamboo Project key
       Default: MCI
    -project-name
       Bamboo Project name
       Default: MettleCI
    -datastage-domain
       DataStage domain
       Default: <empty string>
    -datastage-server
       DataStage server
       Default: <empty string>
    -datastage-project
       DataStage project
       Default: <empty string>
    -compliance-repository
       Bamboo Linked Repository containing Compliance Rules
       Default: Compliance Rules
    -deployment-project
       Bamboo Deployment name
    -E
       Environment names using <ID>=<Name> syntax
       Syntax: -Ekey=value
       Default: {}

Parameter	Required?	Description

Parameter	Required?	Description
Bamboo Connection parameters
-bamboo		URL to access Bamboo
-username		Bamboo Username
-password		Bamboo Password
Continuous Integration parameters
-plan-key		2-5 upper case letter code that Bamboo will use to create a unique identifier for the deployed Continuous Integration plan. For example, `CI`.
-plan-name		The name for the deploy Continuous Integration plan which will be displayed by Bamboo. For example, `Continuous Integration`.
-project-key		2-5 upper case letter code that Bamboo will use to group multiple Bamboo plans. If a project with a matching key exists the Continuous Integration plan will deployed as part of the existing project, otherwise it will be created. Defaults to `MCI`
-project-name		The name of the Bamboo project which the Continuous Integration plan will be deployed under. A Bamboo project is a named group of related Bamboo plans. Defaults to `MettleCI`
-source-repository		Name of the linked source code repository that is used as the DataStage Asset Repository.
-compliance-repository		Name of the linked source code repository that is used as the Compliance Respository. Defaults to `Compliance Rules`
-datastage-install		Name of the DataStage Capability used by the Continuous Integration plan.
-bamboo-shared-credentials		Name of the shared username and password credentials to be used by the Continuous Integration and Deployment plans
DataStage Project parameters
-datastage-domain		DataStage Service tier connection string used by the Continuous Integration plan. This string the same as that used when connecting with the DataStage Designer client.
-datastage-server		DataStage Engine tier host name used by the Continuous Integration plan.
-datastage-project		Logical name of the DataStage Project used by Continuous Integration and Delivery plans. The name of the DataStage project which will be deployed will be `<logical name>_ci` during Continuous Integration or `<logical name >_<environment id>` during Continuous Delivery deployments.
Continuous Delivery parameters
-deployment-project		Name of the Bamboo Deployment Project connected to the Continuous Integration plan. This is required if at least one E parameter is specified.
-E <environment id>=<environment name>		Describes the ID and Name of a Bamboo Deployment Environment. For example, Eqa=Quality Assurance where `qa` is the environment id for an environment called `Quality Assurance`. Repeating this parameter once for every Continuous Deployment environment you need for you DataStage Project.

Post Deployment

Minimal Git Repository Structure

The attached zip file contains the minimal directory structure and files required in the DataStage Asset Repository in order to successfully execute the deployed CI/CD pipeline. Please note that the var.ci file will need to be copied and renamed to var.<environment id> for each environment specified using the -E <environment id>=<environment name> switch.

Manual Bamboo Configuration

There is a limitation within Bamboo which prevents MettleCI from correctly setting shared credentials used by custom Bamboo Tasks, including those which are included as part of MettleCI. Therefore, MettleCI will deploy build and deployment plans in a disabled state with the expectation that MettleCI users will manually set Shared Credentials for custom tasks before enabling the build and deployment plans for use.

Upon successful execution of the MettleCI plan deployment command, a list of tasks which need updating will be printed to the terminal. For each of these Bamboo plan tasks, complete the following procedure:

Locate the specified Bamboo Task in the Continuous Integration build plan or each Environment deployment plan
Select the Shared username and password credentials to match those specified by the -bamboo-shared-credentials switch:
Click Save:

Once all custom Bamboo Tasks have been updated to use the desired shared credentials, the Continuous Integration plan can be enabled:

Set the Information Server instance used by each Environment

The MettleCI command for deploying Bamboo build and deployment plans assumes that all Bamboo Deployment Environments will deploy to the same Information Server instance and engine. To associate an environment with a different Information Server Engine or Services tier, modify the appropriate variables associated with the Bamboo Deployment Environment:

MettleCI Documentation

Generating MettleCI Bamboo Plans (Deprecated)