Document toolboxDocument toolbox

Does Mettle CI support Git branching/merging functionality?

For non-binary, non-proprietary filesystem assets, yes it does. Git supports this by default.

For DataStage Jobs, MettleCI doesn't because DataStage doesn't.

This question is asked by almost every potential MettleCI customer and, usually, it arises from either:

  • a team that is already using a combination of branch-based development lifecycle practices and a non-Git version control system that supports "pessimistic locking"; and / or

  • having knowledge of how Git is used in other development contexts (e.g. building Java apps).

In both cases, it is natural for people to seek parallels in the Git-plus-DataStage context that MettleCI introduces.

However, the ability to automatically merge two code blocks safely (including the detection of granular conflicts) relies on that code being programmatically parsable. Unlike Java source code, for instance, DataStage Jobs are expressed as graph structures. This means that the activity of merging two versions of a Job will always demand human review and human judgements about:

  • which aspects of the two graphs might be in conflict; and

  • what should be removed, retained or changed to achieve a single Job that still meets the intended function of the two versions.

The fundamental questions we always explore with customers to resolve this topic are:

  1. What is the real-world* DataStage development problem that one is seeking to solve with branching and merging?

  2. Why is this problem arising in your team, solution or platform?

  • currently occurring or is reasonably expected to occur, and at a scale or frequency that demands an automated solution.

There is no simple answer to branching and merging DataStage Jobs at scale. We therefore suggest teams start by focusing on where MettleCI already provides the opportunity to achieve relatively higher-value and more tangible improvements to a team's DataStage development practices and time-to-delivery. These alone will justify the investment and improve ETL outcomes.

© 2015-2024 Data Migrators Pty Ltd.