MettleCI Platform Components

The following high-level MettleCI architecture shows the key software components to be installed, and their communications between hosts. 

The key components are:

• Atlassian Bamboo Server, or equivalent
• Atlassian Bitbucket Server, or equivalent
• Atlassian Jira Server, work item management solution, , or equivalent
• IBM InfoSphere Information Server Client (the version should be the same as your DataStage installation) for connection to your chosen non-production IIS environment's Services tier
• MettleCI Scheduler instance (optional)
• MettleCI Wallboard server instance (optional, but recommended)

Some points to note:

• MettleCI works with all Information Server deployment topologies, including High Availability, Grid, and Cluster environments, For clarity, this diagram shows each tier residing on its own, dedicated host.

• Bamboo and associated Remote Agents aren't required if you only wish to use MettleCI's Unit Test Automation, Compliance and Git Check-In functions, as these are all accessible directly from the MettleCI Workbench.

Infrastructure

Servers

MettleCI requires a single, dedicated physical or virtual server with the following specification to act as the MettleCI Host:

Some points to note:

• MettleCI works with all Information Server deployment topologies, including High Availability, Grid, and Cluster environments, For clarity, this diagram shows each tier residing on its own, dedicated host.
• Application Lifecycle Management components can be co-hosted with the other tools on the MettleCI Host or located on another host (on-premise, cloud or SaaS).

Infrastructure Requirements

MettleCI Host

MettleCI requires a dedicated physical or virtual server with the following specification to act as the MettleCI Host:

• A recent model 8-Core (minimum) Intel-compatible CPU running at 2.5GHz (minimum)
• 16GB RAM (minimum)
• 250GB (minimum) available high-speed disk, allocated as
  • 100GB for MettleCI components
  • 150GB (estimated) for your assets in Git. Increase this value if you believe your Information Server assets will occupy more space than this.
• Microsoft Windows Server 2012 or 2016 Base (64-Bit) operating system. 
  • The Windows Server 2016 version is preferred as it doesn't have the file path length limitation that is present in version 2012. This can be a factor for some DataStage Projects which have long Category names and deep Category folder structures.
  • This should be a ‘clean’ operating system, which does not contain any co-resident third party software or artefacts remaining from previous software installations.
  • Ensure your Windows version is compatible with the DataStage Client Tools bundled with your Information Server version.  For more information visit https://www-01.ibm.com/support/docview.wss?uid=swg27050442.

Connectivity and Privileges

The MettleCI topology described in this document requires the following network connectivity:

• High-speed network connectivity between the MettleCI Host and your nominated Development and Test Information Server Environments.
• Network connectivity to allow the Information Server Client tools running on the MettleCI Host to communicate over standard, IBM-documented protocols and ports with the Information Server Engine and Service tiers for each environment to which you wish to deploy Releases.
• RDP (Remote Desktop Protocol) access to the MettleCI Host should be enabled
• The MettleCI Host should be permitted to remotely invoke a command shell on your DataStage Engine Tier using SSH. 
• Temporary Window Administrator permissions on the MettleCI Host to deploy MettleCI-related components and configure the necessary Windows services (described elsewhere in this document).
• At least one Service Account to trigger actions on the MettleCI Host with necessary permissions to interact with the Information Server environments.
• Optional: Temporary Window Administrator access on each Developer's Workstation to install and configure the MettleCI custom menus in the DataStage Designer client.  Alternatively, these can be configured manually by Developers using the DataStage Designer client with no elevated privileges.
• Developers, Testers, and any other MettleCI users should be provided with unfiltered HTTP (port 80) and HTTPS (port 443) internet access to the following domains, to enable access to all support resources:
  • *.mettleci.com
  • *.mettleci.io
  • datamigrators.atlassian.net

Software

Software Requirements

The MettleCI commissioning process can be expedited by pre-configuring the MettleCI Host with the following software components: 

• MettleCI installation media and licence. 
  • Contact your Data Migrators or IBM client representative for access to these.
• Oracle Java Development Kit (version 1.8 preferably latest build)
  • https://www.oracle.com/technetwork/java/javase/downloads/index.html
• Git client (latest version)
  • https://git-scm.com/downloads
• NodeJS (v10)
  • https://nodejs.org/en/download/
• NGINX for windows (v1.15.3 onwards)
  • https://nginx.org/en/docs/windows.html
• NSSM (v2.24)
  • https://nssm.cc/download
• PostgreSQL (v9.6 latest build)
  • https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
• IBM Information Server Client media available from the MettleCI Host, either from a local disk or shared network drive.
  • Available through your IBM Passport Advantage account
  • Note: This software should NOT be pre-installed prior to the MettleCI commissioning process. It is essential that during commissioning Atlassian Bamboo is installed prior to the IBM Information Server Client, due to the Information Server Client's manipulation on the Path environment variable during its installation. The DataStage client will be installed and its connection to your Information Server instance verified during MettleCI commissioning.

Optionally, the MettleCI Host makes use of the following optional Information Server components:

• Optional: IBM Information Server Information Governance Catalog ('IGC') installed, configured, and available.  MettleCI uses IGC to determine job lineage when generating execution schedules for MettleCI's bundled scheduling tool.  You will not require IGC if you are planning on using your existing job scheduling approach for Continuous Integration (e.g. DataStage Job Sequences, Shell scripts, Control-M, etc.) 
• Optional: IBM Information Server Operations Console installed, configured, and available.  MettleCI uses the Operations Console API for authentication, but can be configured to utilise other authentication services  if this is not available.  Users are required to manually configure their email address in the MettleCI Workbench as part of a one-step registration process during their first login.  This enables MettleCI to identify their subsequent activity in the Bitbucket Git repository.  

Firewall Rules

MettleCI Host

Atlassian Bamboo

MettleCI Bamboo

7990 (HTTP)

7999 (SSH) 

Git source code repository.  This instance uses a single MettleCI Bitbucket Plugin which enables the visualisation of a DataStage job's canvas in Bitbucket's source code preview.

• CPU running at 2.5GHz (minimum)
• 16GB RAM (minimum)
• 250GB (minimum) available high-speed disk, allocated as follows:
  • 100GB for MettleCI components
  • 150GB (estimated) for your assets in Git. Increase this value if you believe your Information Server assets will occupy more space than this.
• Microsoft Windows Server 2016* Base (64-Bit) operating system.
  
  • This should be a ‘clean’ operating system which does not contain any co-resident third party software or artefacts remaining from previous software installations.
  • Ensure your Windows version is compatible with the DataStage Client Tools bundled with your Information Server version.  For more information visit https://www-01.ibm.com/support/docview.wss?uid=swg27050442.
• Able to support two concurrent RDP sessions
• Provides a mechanism to easy transfer files (e.g. application installation files, log files) between the installer's computer and this host.

* Window Server 2012 is a fall-back option but comes with operating system restrictions regarding filesystem path lengths that should be discussed with your MettleCI consultant prior to delivering this infrastructure.

MettleCI Consultant Personal Computer

The MettleCI expert performing your installation - or supporting your staff to perform the installation - will need either

1. the ability to remotely access the hosts shown in the MettleCI Platform Components diagram from their company-supplied computer (running a current MacOS or Windows version) via your VPN; or
2. a laptop computer provided by your organisation that allows them to remotely access the MettleCI Platform Components over a secure connection.

User Accounts

MettleCI needs the following dedicated accounts to enable installation activities as well as support on-going operation.

Note: The names given below are simply proposals for reference purposes and can be changed to meet your organisational requirements. If a customer administers its accounts via a centralised repository (e.g. Active Directory) then any accounts specified in the following list with the same name can, of course, be the same account with privileges as necessary to cover multiple roles across multiple environments and components.

MettleCI Host

1. Two Windows user accounts are required for the purpose of installing and administering MettleCI-related components on this host. They must...
   
   1. ideally be named 'mciconfig1' and 'mciconfig2'
   2. have administrator privileges sufficient to
      1. run Windows tools "...as administrator"
      2. install software for use by other users on the host; and
      3. create, remove, start and stop Windows services.
   3. Note: The second account acts to complement the first account in case the owner of the first account becomes unavailable (e.g. sick) or we collectively choose to perform work in parallel. 
2. A Windows user account is required to act as a service account for executing Atlassian and MettleCI services. It must...
   
   1. ideally be named 'mciservice'
   2. have only sufficient privileges to run the services installed by the 'mciconfig#' accounts

DataStage development Engine Tier

1. An operating system user account and corresponding DataStage application account for in-bulk tasks (e.g. CI, CD) performed by MettleCI
   
   1. Ideally named 'mciagent'
   2. Accessible via SSH from both the MettleCI Legacy Client Tier and the MettleCI consultant's computer.
   3. Capable of accessing your DataStage Engine Tier's filesystem such that DataStage can read files and directories generated or modified by this account.
      1. In particular this account will make changes to the contents of your DataStage Projects directory.
   4. Primary group must be 'dstage' (or equivalent as configured for your DataStage platform)
   5. On *nix hosts, this account must be able to source the $DSHOME/.dsenv file.
   6. The DataStage application account must be assigned the Information Server 'Suite Administrator' role.
2. An operating system user account and corresponding DataStage application account for use by MettleCI end-user functions (e.g. MettleCI Workbench)
   
   1. Ideally named 'mciworkb'
   2. Accessible via SSH from both the MettleCI Legacy Client Tier and the MettleCI consultant's computer.
   3. Primary group must be 'dstage' (or equivalent as configured for your DataStage platform)

   4. root must be able to run commands under this user (via sudo) from within a SysVinit service.

   5. On *nix hosts, this account must be able to source the $DSHOME/.dsenv file.
   6. The DataStage application account must be assigned the Information Server 'Suite Administrator' role.

Password Expiry

All MettleCI-related service accounts (both application and operating system) should have passwords that either don't expire or are set to expire at the forecast end of the upgrade initiative. A change to a service account password will require updates to MettleCI configuration to re-enable the MettleCI functions that rely on that service account. Unplanned service account password changes are highly likely to cause unnecessary delay to your upgrade initiative.

Firewall Rules

Public Internet Access

Ideally, developers, testers, and any other MettleCI users should be provided with unfiltered HTTP (port 80) and HTTPS (port 443) internet access to the following domains for installation purposes only:

• *.mettleci.com (for access to the self-service MettleCI Report Card)
• *.mettleci.io (documentation and software downloads)
• datamigrators.atlassian.net (support portal)

Component Connections

The following table describes which interactions take place between the software components listed above, and which functions those interactions support.  Note that if a component or protocol isn't specified then it’s either irrelevant to MettleCI, or MettleCI can re-use whatever your infrastructure dictates.

MettleCI Host

Atlassian suite

(JIRA, Bitbucket, 

Bamboo, Crowd, 

Confluence)

instance with the Information Server 'Suite Administrator' role which MettleCI uses to access your IBM InfoSphere Information Server instance.

User Accounts

MettleCI needs the following dedicated user accounts to enable installation activities as well as support on-going operation. Note that the names given below are simply proposals for reference purposes and can be changed to meet your organisational requirements.

MettleCI Host

1. Two Windows user accounts 'mciconfig1' and 'mciconfig2'
   1. A pair of dedicated Windows operating system user account with Administrator privileges necessary to install and configure MettleCI software and services on the MettleCI Host.
   2. This accounts can be disabled/removed once the installation and configuration of MettleCI has been completed.
   3. The second accounts acts to complement the first account, in case the owner of the first account becomes unavailable (e.g. sick) or we collectively choose to perform work in parallel. 
   4. NOTE: That the MettleCI Host will need the ability to support two concurrent RDP sessions  
2. A single Windows user account 'mciservice'
   1. A service account for executing Atlassian and MettleCI services
   2. Requires fewer privileges than the 'mciconfig-' accounts.
   3. This account requires the ability to create Windows filesystem symbolic links ('symlinks') to support MettleCI Automated Scheduler functionality.

DataStage Engine Tier

1. A single Operating system user account 'mciagent'
   1. Accessible via SSH from a remote location
   2. Capable of accessing your DataStage Engine Tier's filesystem such that DataStage can read files and directories generated or modified by this account.
      1. In particular, this account will make changes to the contents of the DataStage Projects directory (<Information Server base installation path>/IBM/InformationServer/Server/Projects).
2. A single Operating system user account 'mciworkb'
   1. A dedicated operating system user account used for running the MettleCI workbench service on the Engine.
   2. On Unix-based operating systems this could be achieved by making this user a member of the 'dstage' group, on this assumption that this group had the appropriate read/write privileges.  Alternatively, you could use 'dsadm' in place of this account.

Application Accounts

Software

Software Requirements

The MettleCI commissioning process can be expedited by pre-configuring the MettleCI Host with the following software components: 

• MettleCI installation media and licence. 
  • Contact your Data Migrators or IBM client representative for access to these.
• Oracle Java Development Kit (version 1.8 preferably latest build)
  • https://www.oracle.com/technetwork/java/javase/downloads/index.html
• Git client (latest version)
  • https://git-scm.com/downloads
• NodeJS (v10)
  • https://nodejs.org/en/download/
• NGINX for windows (v1.15.3 onwards)
  • https://nginx.org/en/docs/windows.html
• NSSM (v2.24)
  • https://nssm.cc/download
• PostgreSQL (v9.6 latest build)
  • https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
• IBM Information Server Client media available from the MettleCI Host, either from a local disk or shared network drive.
  • Available through your IBM Passport Advantage account
  • Note: This software should NOT be pre-installed prior to the MettleCI commissioning process. It is essential that during commissioning Atlassian Bamboo is installed prior to the IBM Information Server Client, due to the Information Server Client's manipulation on the Path environment variable during its installation. The DataStage client will be installed and its connection to your Information Server instance verified during MettleCI commissioning.

Optionally, the MettleCI Host makes use of the following optional Information Server components:

• Optional: IBM Information Server Information Governance Catalog ('IGC') installed, configured, and available.  MettleCI uses IGC to determine job lineage when generating execution schedules for MettleCI's bundled scheduling tool.  You will not require IGC if you are planning on using your existing job scheduling approach for Continuous Integration (e.g. DataStage Job Sequences, Shell scripts, Control-M, etc.) 
• Optional: IBM Information Server Operations Console installed, configured, and available.  MettleCI uses the Operations Console API for authentication, but can be configured to utilise other authentication services  if this is not available.  Users are required to manually configure their email address in the MettleCI Workbench as part of a one-step registration process during their first login.  This enables MettleCI to identify their subsequent activity in the Bitbucket Git repository.  

Security

• All services on the MettleCI server are accessed via a reverse proxy which can be configured to use HTTP (typically on port 80) or HTTPS (typically on port 443). 
• If you wish to configure MettleCI to use HTTPS you will need to provide a certificate.  We will work with you to describe how your certificate renewal process will work with MettleCI's components.
• MettleCI Workbench uses your existing Information Server authentication scheme, operating via the Operations Console, so users login to Workbench using the same credentials they use for DataStage designer.

Licences

• If you licence Information Server under a model that requires per-seat licences, please ensure that at least three seat licences are reserved for MettleCI:
  • 1 x MettleCI Server Service Account
  • 2 x MettleCI Developers (only required for the duration of the MettleCI commissioning process)
• Evaluation: An evaluation licence for MettleCI will be granted for the duration of the evaluation.  Similarly, licences for all Atlassian components will be supplied gratis by Data Migrators for the duration of the evaluation.
• Production: Your purchase entitles you to 12 months of MettleCI updates and electronic support.  Your licence expiry date can be checked in the MettleCI workbench, or in the 'MettleCI' section of your Atlassian Bamboo instance's 'Settings' page.  Bamboo will display a warning message when your licence is within 30 days of expiry, and a notification message when your licence has expired.