...
...
Introduction
...
Introduction
This section describes the hardware, software,
...
and network configuration that needs to be in place before MettleCI can be deployed and configured in your environment.
...
The deployment topology of a MettleCI platform, whether being used for evaluation or production purposes, has been deliberately designed to be as simple as possible. The biggest difference between evaluation and production deployments of MettleCI is the selection of software components residing on the dedicated MettleCI Host that would be better deployed to separate servers. Alternatively, some components of the MettleCI stack could be delivered by integrating with existing solutions already in place in your organisation. While MettleCI is easily integrated with Customers' existing Jira, Git, and Bamboo instances, describing the full range of integration options available is beyond the scope of this document.
The delivered architecture will focus on evaluation purposes only, and has been designed for ease and speed of deployment. The evaluation will validate which parts of this architecture could be re-used for a MettleCI production deployment, and which may need to be re-deployed or reconfigured to meet your organisational requirements.
Whilst not representing our recommended production deployment of MettleCI, this MettleCI architecture assembles all of the software forming MettleCI's DevOps platform for Information Server into a single, pre-configured, easily deployed package. This optimises the time spent on the evaluation process.
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
|
| Info |
|---|
See also |
...
Rapid DataStage Upgrade - Technical Prerequisites for a description of the Rapid DataStage Upgrade technical pre-requisites for MettleCI. |
MettleCI Platform Components
The following high-level MettleCI architecture shows the key software components to be installed, and their communications between hosts.
| Gliffy | ||||||
|---|---|---|---|---|---|---|
|
...
|
...
|
...
The key components are:
- Atlassian Crowd (Single Sign-On for Atlassian and MettleCI Wallboard)
- Atlassian Bamboo Server, with MettleCI extensions
- Atlassian Bitbucket Server, with MettleCI extension
- Atlassian Jira Server, work item management solution
- Generic MettleCI Bamboo Plans, supporting all MettleCI capabilities. These will be customised during MettleCI commissioning to reflect your environmental and DataStage Project needs.
- IBM InfoSphere Information Server Client (version should be the same as your DataStage installation) for connection to your chosen non-production IIS environment's Services tier
- MettleCI Scheduler instance (previously 'Azkaban')
- MettleCI Wallboard server instance
- The data storage technology necessary for each of the software components. In this case, PostgreSQL has been selected for all Atlassian components. Other components use simple filesystem storage.
- A single Bamboo Remote Agent licence, removing the restriction on the size of Bamboo Plans, and hence unlocking full MettleCI CI/CD functionality
- An NGINX web server acting as reverse proxy, so only a single port is required to access all MettleCI web interfaces.
- A default installation of NodeJS v11 on the MettleCI host to run MettleCI Wallboard.
Interaction between those components described in the diagram are...
- Developers use a web browser to interact directly the MettleCI Workbench application served from the DataStage Engine tier. Note that this is a lightweight, JavaScript application that runs in the developer's browser. Workbench functionality is most conveniently invoked using the MettleCI Custom Menus installed in the DataStage Designer client. An installer for the menus is accessible from within the Workbench application, and will install the menus on the Developer's Workstation without requiring Windows Administrator access. Alternatively, menus can be configured manually by Developers in the DataStage Designer client in a few minutes, requiring no elevated privileges. The browser connection to the MettleCI Workbench application is also used for downloading and uploading Unit Test data files to and from the filesystem on the DataStage Engine Tier.
- Developers' regular Information Server client tools (such as DataStage Designer) require direct access to the relevant ports on the Services tier.
- Developers interact with each of the applications running on the MettleCI Host via the reverse proxy running on port 80 (for HTTP) or port 443 (for HTTPS).
- The applications running on the MettleCI Host interact with your Information Server Services Tier to perform queries and submit requests. This interaction uses the normal Information Server ports configured during your Information Server installation.
- The MettleCI Host interacts with Information Server Engine Tier in a number of contexts:
- The MettleCI Workbench performs a check in of DataStage assets and Unit Tests directly to the Bitbucket Git repository
- The MettleCI Workbench will pull an up-to-date view of the compliance rule library from the Bitbucket Git repository on each Compliance run
- To enable a Jira search form the check-in page on the Workbench the IIS Engine Tier must be able to communicate to the MettleCI Host on port 80 (for HTTP) or port 443 (for HTTPS).
- Bamboo's automated Builds create packages of ship-able code called Releases. Bamboo 'Deployment Projects' are configured to deploy selected Releases to downstream Test and Production environments. This will require that the DataStage Client running on the MettleCI Host has network access to each environment to which you wish to deploy Releases.
...
|
In-scope Hosts
Developer's Workstation: Where the Windows DataStage Designer client is typically run
Application Lifecycle Management Tools: One or more hosts running Work Item Management, Git, and Build services
Information Server Development Environment: Your development instance of Information Server, which may be deployed in any topology, and on any number of hosts.
Other Information Server Environment(s): Downstream Information Server environments, including testing and (optionally) Production. These environments can be MettleCI deployment targets without requiring the deployment of any MettleCI components.
MettleCI Agent Host: A MettleCI-dedicated Windows server hosting an IBM DataStage Client tier which is used by your Build system's agent, in conjunction with the MettleCI Command Line Interface, to automate build and deployment activities.
| Info |
|---|
NOTE
|
- Bamboo and associated Remote Agents aren't required if you only wish to use MettleCI's Unit Test Automation, Compliance and Git Check-In functions, as these are all accessible directly from the MettleCI Workbench.
Infrastructure
Servers
MettleCI requires a single, dedicated physical or virtual server with the following specification to act as the MettleCI Host:
- A recent model 8-Core (minimum) Intel-compatible CPU running at 2.5GHz (minimum)
- 16GB RAM (minimum)
- 250GB (minimum) available high-speed disk, allocated as
- 100GB for MettleCI components
- 150GB (estimated) for your assets in Git. Increase this value if you believe your Information Server assets will occupy more space than this.
- Microsoft Windows Server 2012 or 2016 Base (64-Bit) operating system.
- The Windows Server 2016 version is preferred as it doesn't have the file path length limitation that is present in version 2012. This can be a factor for some DataStage Projects which have long Category names and deep Category folder structures.
- This should be a ‘clean’ operating system, which does not contain any co-resident third party software or artefacts remaining from previous software installations.
- Ensure your Windows version is compatible with the DataStage Client Tools bundled with your Information Server version. For more information visit https://www-01.ibm.com/support/docview.wss?uid=swg27050442.
Connectivity and Privileges
The MettleCI topology described in this document requires the following network connectivity:
- High-speed network connectivity between the MettleCI Host and your nominated Development and Test Information Server Environments.
- Network connectivity to allow the Information Server Client tools running on the MettleCI Host to communicate over standard, IBM-documented protocols and ports with the Information Server Engine and Service tiers for each environment to which you wish to deploy Releases.
- RDP (Remote Desktop Protocol) access to the MettleCI Host should be enabled
- The MettleCI Host should be permitted to remotely invoke a command shell on your DataStage Engine Tier using SSH.
- Temporary Window Administrator permissions on the MettleCI Host to deploy MettleCI-related components and configure the necessary Windows services (described elsewhere in this document).
- At least one Service Account to trigger actions on the MettleCI Host with necessary permissions to interact with the Information Server environments.
- Optional: Temporary Window Administrator access on each Developer's Workstation to install and configure the MettleCI custom menus in the DataStage Designer client. Alternatively these can be configured manually by Developers using the DataStage Designer client with no elevated privileges.
- Developers, Testers, and any other MettleCI users should be provided with unfiltered HTTP (port 80) and HTTPS (port 443) internet access to the following domains, to enable access to all support resources:
- *.mettleci.com
- *.mettleci.io
- datamigrators.atlassian.net
Software
Software Requirements
The MettleCI commissioning process can be expedited by pre-configuring the MettleCI Host with the following software components:
- MettleCI installation media and licence.
- Contact your Data Migrators or IBM client representative for access to these.
- Oracle Java Development Kit (version 1.8 preferably latest build)
- Git client (latest version)
- NodeJS (v10)
- NGINX for windows (v1.15.3 onwards)
- NSSM (v2.24)
- PostgreSQL (v9.6 latest build)
- IBM Information Server Client media available from the MettleCI Host, either from a local disk or shared network drive.
- Available through your IBM Passport Advantage account
- Note: This software should NOT be pre-installed prior to the MettleCI commissioning process. It is essential that during commissioning Atlassian Bamboo is installed prior to the IBM Information Server Client, due to the Information Server Client's manipulation on the Path environment variable during its installation. The DataStage client will be installed and its connection to your Information Server instance verified during MettleCI commissioning.
Optionally, the MettleCI Host makes use of the following optional Information Server components:
- Optional: IBM Information Server Information Governance Catalog ('IGC') installed, configured, and available. MettleCI uses IGC to determine job lineage when generating execution schedules for MettleCI's bundled scheduling tool. You will not require IGC if you are planning on using your existing job scheduling approach for Continuous Integration (e.g. DataStage Job Sequences, Shell scripts, Control-M, etc.)
- Optional: IBM Information Server Operations Console installed, configured, and available, MettleCI uses the Operations Console API for authentication, but can be configured to utilise other authentication services if this is not available. Users are required to manually configure their email address in the MettleCI Workbench as part of a one-step registration process during their first login. This enables MettleCI to identify their subsequent activity in the Bitbucket Git repository.
Firewall Rules
...
MettleCI Host
...
Atlassian Bamboo
...
MettleCI Bamboo
...
7990 (HTTP)
7999 (SSH)
...
Git source code repository. This instance uses a single MettleCI Bitbucket Plugin which enables the visualisation of a DataStage job's canvas in Bitbucket's source code preview.
...
Interfacing MettleCI to Information Server via whichever port you are currently using for DataStage clients. This Designer Client is exclusively for MettleCI automated use only. Internal port is set based on customer standards, and/or as required by IBM.
...
3389 (default)
...
n/a
...
DataStage Development Engine Tier
...
MettleCI Workbench
...
8080 & 8081
...
8080 & 8081
...
Component Connections
The following table describes which interactions take place between software components the software components listed above, and which functions those interactions support. Note that if a component or protocol isn't specified then it’s either irrelevant to MettleCI, or MettleCI can re-use whatever your infrastructure dictates.
...
MettleCI Host
...
Atlassian suite
(JIRA, Bitbucket,
Bamboo, Crowd,
Confluence)
...
- Check-In UI
- Compliance UI
- Release Deployment
- Build Summary Results (Automated Testing/Continuous Integration)
- GIT repository viewer
...
- Automated execution monitoring
- Ad hoc execution
- Execution Failure Recovery
...
- Solution health monitoring
- Build/Delivery metric monitoring
...
- Filesystem asset version control
- Non Information Server (ISX) asset version control
...
- Administration of Bamboo-triggered and supporting processes
- Administration of Azkaban Software
- Detailed debug log access
...
MettleCI Host
...
- Check-In
- Compilation
- Compliance
- Deployment (CI and beyond)
- Execution
- Testing
...
- Check-In
...
- Sprint Health Metrics
- Build/Deployment Health Metrics
...
- Check-In
- Compilation
- Deployment
- Compliance
- Execution
- Testing
...
- Deployment (via SCP)
- Script Execution (via SSH)
...
- Check-In
- Compilation
- Deployment
- Compliance
- Execution
- Testing
...
The MettleCI solution will use the same ports and protocols that the customer uses for its standard installation of the DataStage Client on end-user hardware.
...
- Solution health monitoring
...
- Execution
- Testing
User Accounts
MettleCI needs the following dedicated user accounts to enable installation activities as well as support on-going operation. Note that the names given below are simply proposals for reference purposes, and can be changed to meet your organisational requirements.
MettleCI Host
- Two Windows user accounts 'mciconfig1' and 'mciconfig2'
- A pair of dedicated Windows operating system user account with Administrator privileges necessary to install and configure MettleCI software and services on the MettleCI Host.
- This accounts can be disabled/removed once the installation and configuration of MettleCI has been completed.
- The second accounts acts to complement the first account, in case the owner of the first account becomes unavailable (e.g. sick) or we collectively choose to perform work in parallel.
- NOTE: That the MettleCI Host will need the ability to support two concurrent RDP sessions
- A single Windows user account 'mciservice'
- A service account for executing Atlassian and MettleCI services
- Requires fewer privileges than the 'mciconfig-' accounts.
- This account requires the ability to create Windows filesystem symbolic links ('symlinks') to support MettleCI Automated Scheduler functionality.
DataStage Engine Tier
- A single Operating system user account 'mciagent'
- Accessible via SSH from a remote location
- Capable of accessing your DataStage Engine Tier's filesystem such that DataStage can read files and directories generated or modified by this account.
- In particular, this account will make changes to the contents of the DataStage Projects directory (<Information Server base installation path>/IBM/InformationServer/Server/Projects).
- A single Operating system user account 'mciworkbench'
- A dedicated operating system user account used for running the MettleCI workbench service on the Engine.
- On Unix-based operating systems this could be achieved by making this user a member of the 'dstage' group, on this assumption that this group had the appropriate read/write privileges. Alternatively you could use 'dsadm' in place of this account.
Application Accounts
A single Information Server application user account on your DataStage Development instance with the Information Server 'Suite Administrator' role which MettleCI uses to access your IBM InfoSphere Information Server instance.
Security
- All services on the MettleCI server are accessed via a reverse proxy which can be configured to use HTTP (typically on port 80) or HTTPS (typically on port 443).
- If you wish to configure MettleCI to use HTTPS you will need to provide a certificate. We will work with you to describe how your certificate renewal process will work with MettleCI's components.
- MettleCI Workbench uses your existing Information Server authentication scheme, operating via the Operations Console, so users login to Workbench using the same credentials they use for DataStage designer.
Licences
...
- 2 x MettleCI Developers
- 1 x MettleCI Server Service Account
...
|
Connections
The MettleCI Workbench application running on your DataStage Engine tier needs to performs a commit to your Git platform.
The MettleCI Workbench application running on your DataStage Engine tier needs to perform a dynamic lookup of Work items when displaying the Git Commit page.
The Developer Workstation provides data engineers with access to the ALM tools' user interfaces via a supported web browser. This should cover...
the Git host,
the Work item Management host, and
the Build Host
The Developer Workstation requires regular DataStage client tier access to the development environment's ...
DataStage Engine tier, and
DataStage Services tier.
Optional link based on your requirements.
Your Build system performs its duties via its agent installed on the MettleCI Host
The MettleCI Host requires regular access to the development environment's ...
DataStage Engine tier, and
DataStage Services tier
The MettleCI Host requires regular access to the downstream test environments' DataStage Engine and Services tiers, to affect automated deployment.
See also
| Child pages (Children Display) |
|---|
...