There will always be times when, despite your team’s best efforts, critical defects will arise in a production environment whose business impact will require a timely response. Many teams respond to this sidestepping all of their existing governance processes in a panic and forcing a hurriedly constructed patch into their production environment with little thought to testing, or how those changes will be integrated back into their main development codebase.

MettleCI ships with a set of example CI/CD build pipelines supporting the most commonly used build systems. These examples also include pipeline which demonstrate how to…

• implement a well-governed Production hotfix process without having to omit the most important testing steps, and

• incorporate those changes back into your development codebase.

This section describes the philosophy and process behind these examples.

Open

When you encounter a defect in production you have three options:

Fix in Development and promote as normal.

Open

In this scenario the Developer will use diagnostic information derived from Production defects to identify the root cause of the issue and fix it in the Development environment. From there on the testing and deployment of your changes follows that of a regular delivery process. Adopting this approach does not involve omitting downstream testing processes in intermediate environments (such as those used for Integration or Performance testing).

The MettleCI deployment process will synchronise the complete configuration of your Git repository to the target environment (Production). For this reason, the use of this approach requires that no commits have been made to the Git repository since the repository’s codebase was promoted to Production. This may mean that the development of a hotfix may require pausing the development of new features (or of other, non-time critical defects) in your shared Development environment.

A benefit of this approach is that Development continues as normal and there is no requirement for a subsequent remediation process, such as merging code branches, for example. Because you’re not creating code branches your Git repository's commit history also remains simple to understand and communicate.

Re-deploy a previously approved version of Production

In this scenario a previously released production codebase is restored to the Production environment by simply re-executing its deployment plan from your build system. This approach offers the lowest effort, cost, and risk to restore production to a known (working) state. Resolution of the defect can then take place in a non-time critical fashion in your development environment as usual. This works well for defects introduced by code changes, but may or may not resolve issues introduced by external changes, unless those too can be reverted.

Develop a hotfix

This is the most complex, sophisticated, and flexible of the three scenarios and involves the use a git code branch to support the delivery of time-critical changes to Production whilst avoiding the need to halt ongoing development of new feature and non-time critical defects in your development codebase.

Preparing to develop a hotfix

Hotfixes are developed independently of your main development branch. To prepare for a potential hotfix every deployment to production involves creating a Git branch with a naming scheme that indicates that the branch is intended for the development of a hotfix, and which includes a reference to the production version from which it was created. This naming scheme may differ between build systems, based on their individual capabilities, but in the example shown above the deployment process has deployed production build 43 and created an associated branch named 'Hotfix-43’.

The codebase represented by your hotfix branch is deployed to a dedicated ‘hotfix development’ project which fulfils the same roles as your regular Development environment but is used exclusively for developing hotfixes.

To expedite the development of hotfixes every deployment to Production will also involve, once the hotfix branch has been created, the deployment of that codebase to the Hotfix Development project, labelled ‘DEV_HF’ in the diagram above. The means that development of a hotfix (should one be required) can commence without delay.

Each deployment to production will also involve the deployment of the latest production code configuration to the hotfix development project. The same hotfix development project is used for every hotfix branch, and it is simply updated to mirror whatever software configuration is currently in production. Each time this happens, MettleCI Workbench needs to know which Git branch the hotfix development project currently represents. This is achieved by updating Workbench’s project registration information from your release deployment plan by having it call the workbench set-branch (documentation) command. This is demonstrated by the example hotfix pipelines provided with MettleCI.