MettleCI Platform Components
The following high-level MettleCI architecture shows the key software components to be installed, and and their communications between hosts.
The key components are:
- Atlassian Crowd (Single Sign-On for Atlassian and MettleCI applications)
- Atlassian Jira Server, work item management solution
- Atlassian Bamboo Server, with MettleCI extensions
- Atlassian Bitbucket Server, with MettleCI extensions
- Generic MettleCI Bamboo Server Plans, supporting all MettleCI capabilities. These will be customised during MettleCI commissioning to reflect your environmental and DataStage Project needs.
- IBM InfoSphere Information Server Client (of an appropriate version) for connection to your chosen non-production IIS environment's Services tier
- MettleCI Scheduler instance (previously 'Azkaban')
- MettleCI Wallboard server instance
- The data storage technology necessary for each of the software components. In this case, Postgres has been selected for all Atlassian components. Other components use simple filesystem storage.
- A single Bamboo Remote Agent licence supporting, removing the restriction on the size of Bamboo Plans, and hence unlocking full MettleCI CI/CD functionality
- An NGINX web server acting as reverse proxy, so only a single port is required to access all MettleCI web interfaces.
Interaction between those components described in the diagram are...
- Developers use a web browser to interact directly the MettleCI Workbench application served from the DataStage Engine tier. Note that this is a lightweight, single-page JavaScript application that runs in the developer's browser. Workbench functionality is most conveniently invoked using the MettleCI Custom Menus installed in the DataStage Designer client. An installer for the menus is accessible from within the Workbench application, and will instal the menus on the Developer's Workstation without requiring Windows Administrator access. Alternatively, menus can be configured manually by Developers in the DataStage Designer client in a few minutes, requiring no elevated privileges. The browser connection to the MettleCI Workbench application is also used for downloading and uploading Unit Test data files to and from the filesystem on the DataStage Engine Tier.
- Developers' regular Information Server client tools (such as DataStage Designer) require direct access to the relevant ports on the Services tier.
- Developers interact with each of the applications running on the MettleCI Host via the reverse proxy running on port 80 (for HTTP) or port 443 (for HTTPS).
- The applications running on the MettleCI Host interact with your Information Server Services Tier to perform queries and submit requests. This interaction uses the normal Information Server ports configured during your Information Server installation.
- The MettleCI Host interacts with Information Server Engine Tier in a number of contexts:
- The MettleCI Workbench performs a check in of DataStage assets and Unit Tests directly to the Bitbucket Git repository
- The MettleCI Workbench will pull an up-to-date view of the compliance rule library from the Bitbucket Git repository on each Compliance run
- Bamboo's automated Builds create packages of shippable code called Releases. Bamboo 'Deployment Projects' are configured to deploy selected Releases to downstream Test and Production environments. This will require that the DataStage Client running on the MettleCI Host has network access to each environment to which you wish to deploy Releases.
Some points to note:
- MettleCI works with all Information Server deployment topologies, including High Availability, Grid, and Cluster environments, For clarity, this diagram shows each tier residing on its own, dedicated host.
- Bamboo and its associated agent isn't required if you only wish to use MettleCI's Unit Test Automation, Compliance and Git Check-In functions, as these are all accessible directly from the MettleCI Workbench.
Infrastructure
Servers
MettleCI requires a single, dedicated physical or virtual server with the following specification to act as the MettleCI Host:
- A recent model 4-Core (minimum) Intel-compatible CPU running at 2.5GHz (minimum)
- 16GB RAM (minimum)
- 250GB (minimum) available high-speed disk, allocated as
- 100GB for MettleCI
- 150GB (estimated) for your assets in Git. Increase this value if you believe your assets will occupy more than this.
- Microsoft Windows Server 2012 or 2016 Base (64-Bit) operating system.
- This should be a ‘clean’ operating system, which does not contain any co-resident third party software or artefacts remaining from previous software installations.
- Ensure your Windows version is compatible with the DataStage Client Tools bundled with your Information Server version. For more information visit https://www-01.ibm.com/support/docview.wss?uid=swg27050442.
Connectivity and Privileges
The MettleCI topology describe in this document requires the following network connectivity:
- High-speed network connectivity between the MettleCI Host and your nominated Development and Test Information Server Environments.
- Network connectivity to allow the Information Server Client tools running on the MettleCI Host to communicate over standard, IBM-documented protocols and ports with the Information Server Engine and Service tiers.
- RDP (Remote Desktop Protocol) access to the MettleCI Host should be enabled
- The MettleCI Host should be permitted to remotely invoke a command shell on your DataStage Engine Tier using SSH.
- Temporary Window Administrator permissions on the MettleCI Host to deploy MettleCI-related components and configure the necessary Windows services (described elsewhere in this document).
- At least one Service Account to trigger actions on the MettleCI Host with necessary permissions to interact with the Information Server environments.
- Optional: Temporary Window Administrator access on each Developer's Workstation to install and configure the MettleCI custom menus in the DataStage Designer client. Alternatively these can be configured manually by Developers using the DataStage Designer client with no elevated privileges.
Software
Software Requirements
The MettleCi commissioning process can be expedited by pre-configuring the MettleCI Host with the following software components:
- MettleCI installation media and licence. Contact your Data Migrators or IBM client representative for access to these.
- Oracle Java Development Kit (version 1.8 or greater)
- Git client (latest version)
- IBM Information Server Client media available from the MettleCI Host, either from a local disk or shared network drive.
- Note: This software should NOT be pre-installed prior to the MettleCI commissioning process. It is essential that during commissioning Atlassian Bamboo is installed prior to the IBM Information Server Client, due to the Information Server Client's manipulation on the Path environment variable during its installation. The DataStage client will be installed and its connection to your Information Server instance verified during MettleCI commissioning.
- Optional: IBM Information Server Information Governance Catalog ('IGC') installed, configured, and available. MettleCI uses IGC to determine job lineage when generating execution schedules for MettleCI's bundled scheduling tool. You will not require IGC if you are planning on using your existing job scheduling approach for Continuous Integration (e.g. DataStage Job Sequences, Shell scripts, Control-M, etc.)
- Optional: IBM Information Server Operations Console installed, configured, and available, MettleCI uses the Operations Console API for authentication, but can be configured to utilise other authentication services if this is not available. Users are required to manually configure their email address in the MettleCI Workbench as part of a one-step registration process during their first login. This enables MettleCi to identify their subsequent activity in the Bitbucket Git repository.
| Host | Item | Windows Service Name | Internal Port | External Port | Purpose |
|---|---|---|---|---|---|
MettleCI Host (MettleCI-dedicated Windows server) | Atlassian Bamboo | MettleCI Bamboo | 8085 | 80 / 443 | Build automation server. This instance uses a suite of MettleCI Bamboo Plugins which provide MettleCI's advanced Information Server deployment and build capabilities. |
| Atlassian JIRA | MettleCI JIRA | 8080 | 80 / 443 | Work item management. | |
| Atlassian Bitbucket | MettleCI Bitbucket | 7990 | 80 / 443 | Git source code repository. This instance uses a single MettleCI Bitbucket Plugin which enables the visualisation of a DataStage job's canvas in Bitbucket's source code preview. | |
| Atlassian Crowd | MettleCI Crowd | 8095 | 80 / 443 | Common authentication (Single Sign On). | |
| MettleCI Scheduler | MettleCI Scheduler | 8081 | 80 / 443 | MettleCI Scheduler (previously 'Azkaban') | |
| MettleCI Wallboard | MettleCI Wallboard | 5000 | 80 / 443 | MettleCI Wallboard (previously 'Mozaïk') | |
| NGINX | Reverse Proxy | 80 / 443 | 80 / 443 | Reverse proxy through which all MettleCI Host applications are accessed. | |
| Postgres DB | Postgres DB | n/a | 5432 | Storage service for the Atlassian tools. | |
| DataStage Designer Client | n/a | See relevant IBM documentation | Interfacing MettleCI to Information Server via whichever port you are currently using for DataStage clients. | ||
DataStage Development Engine Tier | MettleCI Workbench | MettleCI Workbench (Windows only) | 8080 & 8081 | Browser-based user interface for Unit Test specification and results, Compliance invocation, and Git check in. Note that diagnostic output is exposed on port 8081. | |
User Accounts
MettleCI needs the following user accounts to enable installation activities as well as support on-going operation:
- A dedicated user account with Administrator privileges necessary to install and configure MettleCI software and services on the MettleCI Host
- A dedicated application service user with the Information Server 'Suite Administrator' role which MettleCI uses to access your IBM InfoSphere Information Server instance.
- A dedicated operating system user account capable of remotely accessing your DataStage Engine Tier's filesystem, required for managing your DataStage solution's filesystem assets it Git.
Depending upon your Information Server authentication configuration, requirements 2 and 3 may be delivered using the same account.
Security
- All services on the MettleCI server are accessed via a reverse proxy which can be configured to use HTTP (typically on port 80) or HTTPS (typically on port 443).
- If you wish to configure MettleCI to use HTTPS you will need to provide a certificate. We will work with you to describe how your certificate renewal process will work with MettleCI's components.
- MettleCI Workbench uses your existing Information Server authentication scheme, operating via the Operations Console, so users login to Workbench using the same credentials they use for DataStage designer.
Licences
- If you licence Information Server under a model that requires per-seat licences, please ensure that at least three seat licences are reserved for the MettleCI PoC:
- 2 x MettleCI Developers
- 1 x MettleCI Server Service Account
- Time-limited licences for MettleCI and the supporting Atlassian software and will be provided Gratis by Data Migrators exclusively for the purposes of the <Client Name In Properties> PoC.
- Evaluation: An evaluation licence for MettleCI will be granted for the duration of the evaluation. Similarly, licences for all Atlassian components will be supplied gratis by Data Migrators for the duration of the evaluation.
- Production: Your purchase entitles you to 12 months of MettleCI updates and electronic support. Your licence expiry date can be checked in the relevant MettleCI section of your Atlassian Bamboo instance. Bamboo will display a message when you licence is close to expiry or has expired.
Two-tier Deployment - Detailed Infrastructure Connections
Note that if a component, protocol, or port isn't specified then it’s either irrelevant to MettleCI, or MettleCI can re-use whatever your infrastructure dictates.
Component A | Component B | Protocol(s) | Default Port(s) | Related MettleCI Function(s) | Notes | ||
|---|---|---|---|---|---|---|---|
| Host | Application | Host | Application | ||||
| Developer Workstation | Web browser | MettleCI Host | Atlassian suite | HTTP(S) | JIRA: 8080 Bitbucket: 7990 Bamboo: 8085 Crowd: 8095 Confluence: 8090 |
| |
| MettleCI Scheduler (Azkaban) | HTTP(S) | Azkaban: 8081 |
| ||||
| MettleCI Wallboard | HTTP(S) | Wallboard: 5000 |
| ||||
| Git client | Bitbucket Git service | SSH / HTTPS | Bitbucket: 7990, 7999 |
| Could be ignored as the Bitbucket UI (HTTPS) provides a lot of support in this area. | ||
| Remote Desktop Client | Remote Desktop Service | RDP | RDP: 3389. Customer to confirm. |
| |||
MettleCI Host | Bamboo | MettleCI Host | Bamboo Remote Agents | Customer to confirm ???? Why Ask The Customer ??? | Bamboo: 54663 |
| |
| MettleCI Wallboard | Information Server Operations Console | HTTP(S) | DataStage Ops Console: 9443 |
| The MettleCI solution will use the ports and protocols mandated by the customer's standards (see IBM IIS documentation) for the database technology that hosts DSODB. | ||
| Bamboo Remote Agents | BitBucket | SSH, HTTP(S) | Bitbucket: 7990, 7999 |
| |||
| MettleCI Wallboard | Atlassian suite | HTTP(S) | JIRA: 8080 Bitbucket: 7990 Bamboo: 8080 |
| The MettleCI Wallboard needs to extract Jira and Bamboo data via their REST APIs to present health metrics to users. | ||
| DataStage Client | Information Server Engine Tier | Information Server processes | Multiple | Customer to confirm |
| The MettleCI solution will use the same ports and protocols that the customer uses for its standard installation of the DataStage Client on end-user hardware. | |
| SSH | SSH server process | SSL | Customer to confirm |
| The MettleCI solution will use the port that the customer's standards mandate for this protocol. | ||
| DataStage Client | Information Server Services Tier | Information Server processes | Multiple | Customer to confirm |
| The MettleCI solution will use the same ports and protocols that the customer uses for its standard installation of the DataStage Client on end-user hardware. | |
| Bamboo Remote Agents | Information Governance Catalog REST API | HTTP(S) | IGC: 9443 |
| |||