MettleCI Platform Components
The following high-level MettleCI architecture shows the key software components to be installed, and their communications between hosts.
| Gliffy | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
The key components are:
- Atlassian Crowd (Single Sign-On for Atlassian and MettleCI Wallboard)
- Atlassian Bamboo Server, with MettleCI extensions
- Atlassian Bitbucket Server, with MettleCI extension
- Atlassian Jira Server, work item management solution
- Generic MettleCI Bamboo Plans, supporting all MettleCI capabilities. These will be customised during MettleCI commissioning to reflect your environmental and DataStage Project needs.
- IBM InfoSphere Information Server Client (version should be the same as your DataStage installation) for connection to your chosen non-production IIS environment's Services tier
- MettleCI Scheduler instance (previously 'Azkaban')
- MettleCI Wallboard server instance
- The data storage technology necessary for each of the software components. In this case, PostgreSQL has been selected for all Atlassian components. Other components use simple filesystem storage.
- A single Bamboo Remote Agent licence, removing the restriction on the size of Bamboo Plans, and hence unlocking full MettleCI CI/CD functionality
- An NGINX web server acting as reverse proxy, so only a single port is required to access all MettleCI web interfaces.
- A default installation of NodeJS v11 on the MettleCI host to run MettleCI Wallboard.
Interaction between those components described in the diagram are...
- Developers use a web browser to interact directly the MettleCI Workbench application served from the DataStage Engine tier. Note that this is a lightweight, JavaScript application that runs in the developer's browser. Workbench functionality is most conveniently invoked using the MettleCI Custom Menus installed in the DataStage Designer client. An installer for the menus is accessible from within the Workbench application, and will install the menus on the Developer's Workstation without requiring Windows Administrator access. Alternatively, menus can be configured manually by Developers in the DataStage Designer client in a few minutes, requiring no elevated privileges. The browser connection to the MettleCI Workbench application is also used for downloading and uploading Unit Test data files to and from the filesystem on the DataStage Engine Tier.
- Developers' regular Information Server client tools (such as DataStage Designer) require direct access to the relevant ports on the Services tier.
- Developers interact with each of the applications running on the MettleCI Host via the reverse proxy running on port 80 (for HTTP) or port 443 (for HTTPS).
- The applications running on the MettleCI Host interact with your Information Server Services Tier to perform queries and submit requests. This interaction uses the normal Information Server ports configured during your Information Server installation.
- The MettleCI Host interacts with Information Server Engine Tier in a number of contexts:
- The MettleCI Workbench performs a check in of DataStage assets and Unit Tests directly to the Bitbucket Git repository
- The MettleCI Workbench will pull an up-to-date view of the compliance rule library from the Bitbucket Git repository on each Compliance run
- To enable a Jira search form the check-in page on the Workbench the IIS Engine Tier must be able to communicate to the MettleCI Host on port 80 (for HTTP) or port 443 (for HTTPS).
- Bamboo's automated Builds create packages of ship-able code called Releases. Bamboo 'Deployment Projects' are configured to deploy selected Releases to downstream Test and Production environments. This will require that the DataStage Client running on the MettleCI Host has network access to each environment to which you wish to deploy Releases.
Some points to note:
- MettleCI works with all Information Server deployment topologies, including High Availability, Grid, and Cluster environments, For clarity, this diagram shows each tier residing on its own, dedicated host.
- Bamboo and associated Remote Agents aren't required if you only wish to use MettleCI's Unit Test Automation, Compliance and Git Check-In functions, as these are all accessible directly from the MettleCI Workbench.
Infrastructure
Servers
MettleCI requires a single, dedicated physical or virtual server with the following specification to act as the MettleCI Host:
- A recent model 8-Core (minimum) Intel-compatible CPU running at 2.5GHz (minimum)
- 16GB RAM (minimum)
- 250GB (minimum) available high-speed disk, allocated as
- 100GB for MettleCI components
- 150GB (estimated) for your assets in Git. Increase this value if you believe your Information Server assets will occupy more space than this.
- Microsoft Windows Server 2012 or 2016 Base (64-Bit) operating system.
- The Windows Server 2016 version is preferred as it doesn't have the file path length limitation that is present in version 2012. This can be a factor for some DataStage Projects which have long Category names and deep Category folder structures.
- This should be a ‘clean’ operating system, which does not contain any co-resident third party software or artefacts remaining from previous software installations.
- Ensure your Windows version is compatible with the DataStage Client Tools bundled with your Information Server version. For more information visit https://www-01.ibm.com/support/docview.wss?uid=swg27050442.
Connectivity and Privileges
The MettleCI topology described in this document requires the following network connectivity:
- High-speed network connectivity between the MettleCI Host and your nominated Development and Test Information Server Environments.
- Network connectivity to allow the Information Server Client tools running on the MettleCI Host to communicate over standard, IBM-documented protocols and ports with the Information Server Engine and Service tiers for each environment to which you wish to deploy Releases.
- RDP (Remote Desktop Protocol) access to the MettleCI Host should be enabled
- The MettleCI Host should be permitted to remotely invoke a command shell on your DataStage Engine Tier using SSH.
- Temporary Window Administrator permissions on the MettleCI Host to deploy MettleCI-related components and configure the necessary Windows services (described elsewhere in this document).
- At least one Service Account to trigger actions on the MettleCI Host with necessary permissions to interact with the Information Server environments.
- Optional: Temporary Window Administrator access on each Developer's Workstation to install and configure the MettleCI custom menus in the DataStage Designer client. Alternatively, these can be configured manually by Developers using the DataStage Designer client with no elevated privileges.
- Developers, Testers, and any other MettleCI users should be provided with unfiltered HTTP (port 80) and HTTPS (port 443) internet access to the following domains, to enable access to all support resources:
- *.mettleci.com
- *.mettleci.io
- datamigrators.atlassian.net
Software
Software Requirements
The MettleCI commissioning process can be expedited by pre-configuring the MettleCI Host with the following software components:
- MettleCI installation media and licence.
- Contact your Data Migrators or IBM client representative for access to these.
- Oracle Java Development Kit (version 1.8 preferably latest build)
- Git client (latest version)
- NodeJS (v10)
- NGINX for windows (v1.15.3 onwards)
- NSSM (v2.24)
- PostgreSQL (v9.6 latest build)
- IBM Information Server Client media available from the MettleCI Host, either from a local disk or shared network drive.
- Available through your IBM Passport Advantage account
- Note: This software should NOT be pre-installed prior to the MettleCI commissioning process. It is essential that during commissioning Atlassian Bamboo is installed prior to the IBM Information Server Client, due to the Information Server Client's manipulation on the Path environment variable during its installation. The DataStage client will be installed and its connection to your Information Server instance verified during MettleCI commissioning.
Optionally, the MettleCI Host makes use of the following optional Information Server components:
- Optional: IBM Information Server Information Governance Catalog ('IGC') installed, configured, and available. MettleCI uses IGC to determine job lineage when generating execution schedules for MettleCI's bundled scheduling tool. You will not require IGC if you are planning on using your existing job scheduling approach for Continuous Integration (e.g. DataStage Job Sequences, Shell scripts, Control-M, etc.)
- Optional: IBM Information Server Operations Console installed, configured, and available. MettleCI uses the Operations Console API for authentication, but can be configured to utilise other authentication services if this is not available. Users are required to manually configure their email address in the MettleCI Workbench as part of a one-step registration process during their first login. This enables MettleCI to identify their subsequent activity in the Bitbucket Git repository.
Firewall Rules
| Host | Item | Windows Service Name | External Port | Internal Port | Comment |
|---|---|---|---|---|---|
MettleCI Host | Atlassian Bamboo | MettleCI Bamboo | n/a | 8085 | Build automation server. This instance uses a suite of MettleCI Bamboo Plugins which provide MettleCI's advanced Information Server deployment and build capabilities. |
| MettleCI Host | Atlassian JIRA | MettleCI JIRA | n/a | 8080 | Work item management. |
| MettleCI Host | Atlassian Bitbucket | MettleCI Bitbucket | 7999 | 7990 (HTTP) 7999 (SSH) | Git source code repository. This instance uses a single MettleCI Bitbucket Plugin which enables the visualisation of a DataStage job's canvas in Bitbucket's source code preview. |
| MettleCI Host | Atlassian Crowd | MettleCI Crowd | n/a | 8095 | Common authentication (Single Sign On). |
| MettleCI Host | MettleCI Scheduler | MettleCI Scheduler | n/a | 8081 | MettleCI Scheduler. |
| MettleCI Host | MettleCI Wallboard | MettleCI Wallboard | n/a | 5000 | MettleCI Wallboard. |
| MettleCI Host | NGINX | Reverse Proxy | 80 / 443 | 5000 / 7990 / 8080 / 8081 / 8085 / 8095 | Reverse proxy through which all MettleCI Host applications are accessed. |
| MettleCI Host | Postgres DB | Postgres DB | n/a | 5432 | Storage service for the Atlassian tools. |
| MettleCI Host | Atlassian Bamboo Local Agent | MettleCI Bamboo | n/a | 54663 | |
| MettleCI Host | DataStage Designer Client | n/a | n/a | As required | Interfacing MettleCI to Information Server via whichever port you are currently using for DataStage clients. This Designer Client is exclusively for MettleCI automated use only. Internal port is set based on customer standards, and/or as required by IBM. |
| MettleCI Host | Windows Remote Desktop | Remote Desktop Services | 3389 (default) | n/a | |
DataStage Development Engine Tier | MettleCI Workbench | MettleCI Workbench | 8080 & 8081 | 8080 & 8081 | Browser-based user interface for Unit Test specification and results, Compliance invocation, and Git check in. Note that these ports are configurable. Port 8081 is required only for the exposure of MettleCI diagnostic output. |
| DataStage Development Engine Tier | DataMigrators access during POC | SSH | 22 | n/a | |
| DataStage Development Engine Tier | Information Server processes | As required | As required | As required | As supported by O/S, configured by customer, and/or required by IBM |
| DataStage Development Engine Tier | SSH server process | As required | As required | As required | As supported by O/S, configured by customer, and/or required by IBM |
| DataStage Development Services Tier | Information Server processes | As required | As required | As required | As supported by O/S, configured by customer, and/or required by IBM |
| DataStage Development Services Tier | Information Server Operations Console | As required | 9443 (default) | 9443 (default) | As supported by O/S, configured by customer, and/or required by IBM |
| DataStage Development Services Tier | Information Governance Catalog REST API | As required | 9443 (default) | 9443 (default) | As supported by O/S, configured by customer, and/or required by IBM |
Component Connections
The following table describes which interactions take place between software components the software components listed above, and which functions those interactions support. Note that if a component or protocol isn't specified then it’s either irrelevant to MettleCI, or MettleCI can re-use whatever your infrastructure dictates.
| Client Host | Client Application | Server Host | Server Application | Protocol(s) | Related MettleCI Function(s) | Comments |
|---|---|---|---|---|---|---|
| Developer Workstation | Web browser | MettleCI Host | Atlassian suite (JIRA, Bitbucket, Bamboo, Crowd, Confluence) | HTTP(S) |
| |
| Developer Workstation | Web browser | MettleCI Host | MettleCI Scheduler (Azkaban) | HTTP(S) |
| |
| Developer Workstation | Web browser | MettleCI Host | MettleCI Wallboard | HTTP(S) |
| |
| Developer Workstation | Git client | MettleCI Host | Bitbucket Git service | SSH / HTTP(S) |
| Could be ignored as the Bitbucket UI (HTTPS) provides a lot of support in this area. |
| Developer Workstation | Remote Desktop Client | MettleCI Host | Remote Desktop Service | RDP |
| |
MettleCI Host | Bamboo | MettleCI Host | Bamboo (Local Agent) | Internal |
| |
| MettleCI Host | Bamboo (Local Agent) | MettleCI Host | BitBucket | SSH (default), HTTP(S) |
| |
| MettleCI Host | MettleCI Wallboard | MettleCI Host | Atlassian suite | HTTP(S) |
| The MettleCI Wallboard needs to extract Jira and Bamboo data via their REST APIs to present health metrics to users. |
| MettleCI Host | DataStage Client | Information Server Engine Tier | Information Server processes | Multiple |
| The MettleCI solution will use the same ports and protocols that the customer uses for its standard installation of the DataStage Client on end-user hardware. |
| MettleCI Host | SSH | Information Server Engine Tier | SSH server process | SSL |
| The MettleCI solution will use the port that the customer's standards mandate for this protocol. |
| MettleCI Host | DataStage Client | Information Server Services Tier | Information Server processes | Multiple |
| The MettleCI solution will use the same ports and protocols that the customer uses for its standard installation of the DataStage Client on end-user hardware. |
| MettleCI Host | MettleCI Wallboard | Information Server Services Tier | Information Server Operations Console | HTTP(S) |
| The MettleCI solution will use the ports and protocols mandated by the customer's standards (see IBM IIS documentation) for the database technology that hosts DSODB. |
| MettleCI Host | Bamboo (Local Agent) | Information Server Services Tier | Information Governance Catalog REST API | HTTP(S) |
|
User Accounts
MettleCI needs the following dedicated user accounts to enable installation activities as well as support on-going operation. Note that the names given below are simply proposals for reference purposes and can be changed to meet your organisational requirements.
MettleCI Host
- Two Windows user accounts 'mciconfig1' and 'mciconfig2'
- A pair of dedicated Windows operating system user account with Administrator privileges necessary to install and configure MettleCI software and services on the MettleCI Host.
- This accounts can be disabled/removed once the installation and configuration of MettleCI has been completed.
- The second accounts acts to complement the first account, in case the owner of the first account becomes unavailable (e.g. sick) or we collectively choose to perform work in parallel.
- NOTE: That the MettleCI Host will need the ability to support two concurrent RDP sessions
- A single Windows user account 'mciservice'
- A service account for executing Atlassian and MettleCI services
- Requires fewer privileges than the 'mciconfig-' accounts.
- This account requires the ability to create Windows filesystem symbolic links ('symlinks') to support MettleCI Automated Scheduler functionality.
DataStage Engine Tier
- A single Operating system user account 'mciagent'
- Accessible via SSH from a remote location
- Capable of accessing your DataStage Engine Tier's filesystem such that DataStage can read files and directories generated or modified by this account.
- In particular, this account will make changes to the contents of the DataStage Projects directory (<Information Server base installation path>/IBM/InformationServer/Server/Projects).
- A single Operating system user account 'mciworkbench'
- A dedicated operating system user account used for running the MettleCI workbench service on the Engine.
- On Unix-based operating systems this could be achieved by making this user a member of the 'dstage' group, on this assumption that this group had the appropriate read/write privileges. Alternatively, you could use 'dsadm' in place of this account.
Application Accounts
A single Information Server application user account on your DataStage Development instance with the Information Server 'Suite Administrator' role which MettleCI uses to access your IBM InfoSphere Information Server instance.
Security
- All services on the MettleCI server are accessed via a reverse proxy which can be configured to use HTTP (typically on port 80) or HTTPS (typically on port 443).
- If you wish to configure MettleCI to use HTTPS you will need to provide a certificate. We will work with you to describe how your certificate renewal process will work with MettleCI's components.
- MettleCI Workbench uses your existing Information Server authentication scheme, operating via the Operations Console, so users login to Workbench using the same credentials they use for DataStage designer.
Licences
- If you licence Information Server under a model that requires per-seat licences, please ensure that at least three seat licences are reserved for the MettleCI PoC:
- 2 x MettleCI Developers
- 1 x MettleCI Server Service Account
- Evaluation: An evaluation licence for MettleCI will be granted for the duration of the evaluation. Similarly, licences for all Atlassian components will be supplied gratis by Data Migrators for the duration of the evaluation.
- Production: Your purchase entitles you to 12 months of MettleCI updates and electronic support. Your licence expiry date can be checked in the MettleCI workbench, or in the 'MettleCI' section of your Atlassian Bamboo instance's 'Settings' page. Bamboo will display a warning message when your licence is within 30 days of expiry, and a notification message when your licence has expired.