Page Comparison

...

A simple but naïve approach to deploying ISX files containing assets with design information would be as follows:

1. Tear down any existing DataStage project in the target environment

2. Create a new DataStage project in the target environment

3. Import all ISX files

4. Compile all jobs in the DataStage project.

This approach is repeatable as the content of the resulting DataStage project will always match the software version represented by the source ISX files. However, performance of this approach is extremely poor and will not allow the resulting CI/CD Pipelines to provide fast feedback to developers. As an example, a DataStage project containing about 500 jobs will typically take over 1.5hrs to perform both the import and compile steps of this deployment approach.

The implementation used by the MettleCI Deployment Command and Incremental Deployment Bamboo Task is logically equivalent to the naïve approach but uses an incremental algorithm to deploy changes in a fast and scalable manor:

1. Source ISX files and the existing DataStage project is analyzed for differences

2. Differences are converted to operations which transform the existing DataStage project to match the source ISX files

3. An impact analysis of changes caused by step 2 is performed and all affected jobs are compiled (eg. if a shared container changed, then all jobs using it will be compiled)

...

A naïve approach would be implemented as follows:

1. Run a MettleCI Incremental Deployment using ISX files with designs

2. Export the entire DataStage project with binaries

The resulting export would have job designs which match the source ISX files while containing the required binaries. Unfortunately, exporting an entire project is too slow and doesn’t scale well as the DataStage project increases in size. A project with about 500 jobs will typically take about 45min to export with binaries.

Instead, the DataStage project export can be performed by MettleCI’s ISX Export Command which can optionally perform an incremental export with or without binaries. Incremental Export reads from a DataStage project and exports its content into a directory of ISX files. If the export directory is empty then Incremental Export will export the entire DataStage project, otherwise only new or changed assets will be exported:

1. Source DataStage project and target export directory containing ISX files are analyzed for differences

2. Differences are converted to operations which update the ISX files in the target export directory

3. When including binaries during export, an impact analysis of changes identified during step 2 is performed and all affected jobs are exported to capture the latest binaries (eg. A shared container changed, so we export the binaries of all jobs which use it)

...

An incremental import using the ISX Import Command is designed to accept a directory of ISX files and ensure that content of the target DataStage project will be updated to match the ISX files. The process is very similar to an Incremental Deployment, the fundamental difference is that it doesn’t automatically compile jobs and is able to deploy binaries from the ISX files:

1. Source ISX files and the existing DataStage project is analyzed for differences

2. Differences are converted to operations which transform the existing DataStage project to match the source ISX files

Incremental Imports can also be configured to exclude design information. When enabled, jobs will not be viewable/editable in the DataStage Designer/Flow Designer and any assets which are not required for execution (such as table definitions, shared containers, etc) will be excluded from the import.

...