...
CI/CD Pipelines will deploy a version of software across multiple environments while subjecting it to harder and harder tests in order to be confident that the software works as intended. As a result, a core component of any CI/CD Pipeline is to be able to repeatably deploy a given version of software to multiple environments without introducing any differences. Differences between software deployed to across different testing environments can trigger random test failures, or worse, software that worked in testing can fail once deployed to production.
MettleCI provides a set of composable deployment tools for performing repeatable and efficient DataStage Project deployments.
...
Source ISX files and the existing DataStage project is are analyzed for differences
Differences are converted to operations which transform the existing DataStage project to match the source ISX files
An impact analysis of changes caused by step 2 is performed and all affected jobs are compiled (eg. if a shared container changed, then all jobs using it will be compiled)
| Gliffy | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Duration of a deployment depends on the number of changes rather than the size of project. For a typical CI/CD Pipeline that is triggered per commit, DataStage project deployments take a few minutes.
...
The resulting export would have job designs which match the source ISX files while containing the required binaries. Unfortunately, exporting an entire project is too slow and doesn’t scale well as the DataStage project increases in size. A project with about 500 jobs will typically take about 45min 45 minutes to export with binaries.
Instead, the DataStage project export can be performed by MettleCI’s ISX Export Command which can optionally perform an incremental export with or without binaries. Incremental Export reads from a DataStage project and exports its content into a directory of ISX files. If the export directory is empty then Incremental Export will export the entire DataStage project, otherwise only new or changed assets will be exported:
...
Source ISX files and the existing DataStage project is are analyzed for differences
Differences are converted to operations which transform the existing DataStage project to match the source ISX files
...