Page Comparison

...

The core principles driving S2PX’s design, in no particular order, are…

• S2PX prioritises the generation of a functionally-accurate Parallel Jobs above all else.

• S2PX aims to provide support the Parallel implementation of as much coverage of Server Job functionality as possible.

• S2PX aims to generate Parallel Job designs with a visual appearance that, as far as possible, matches that of their Server equivalents.

• S2PX design decisions have been informed by a detailed analysis of >150K real-world Server jobs submitted by a broad range of IBM’s DataStage customers from around the world. This evidence alone is used to form our view of the DataStage Server Job landscape.

• S2PX does not automatically convert DataStage BASIC routines into Parallel engine compatible routines, although it does provide facilities to make that processes easier.

• S2PX uses your solution’s design-time information to inform its conversion. Runtime logs are not used to inform conversion decision.

• S2PX generates Parallel Jobs with every stage set to run in sequential mode (for good reasons)

• S2PX converts Hashed files to DRS stages (for good reasons)

• S2PX runs in situ, meaning it exports Server jobs from your existing DataStage environment and delivers automatically-generated Parallel jobs into that same environment. Migrating those newly-generated Parallel jobs to an upgraded environment (e.g. DataStage v11.5 to v11.7, or even to Cloud Pak for Data) is achieved using other tools.

...

If you were to undertake the conversion of a Server Job to a Parallel Job without the use of an automation capability the list of steps you would need to follow would be daunting:

1. Generate a new Parallel Project with the necessary settings

2. For each Server Job…

   1. Decompose the Server Job into ‘sections’ by identifying synchronisation stages where the Server Job must be split

   2. Create a new Parallel Job for each identified Server Job section.

   3. Create a Job Sequence to orchestrate the execution of the new Parallel Jobs, respecting appropriate ordering and trigger conditions

3. For each new Parallel Job…

   1. Ensure Parameters and Parameter Set Values have been propagated from the source Server Job to the Parallel Job

   2. Identify Server-only design patterns in the source Job and transcribe into their Parallel equivalents. e.g.

      1. Server Transformer Lookups to Parallel Lookups Stages,

      2. Server Transformer Rejects to Parallel reject from downstream stage(s),

      3. etc.

   3. Identify whether the job’s design patterns will need to be implemented as multiple Parallel jobs, and if so…

      1. Identify the interfacing stage(s) between each of those Parallel jobs

      2. Ensure each Parallel job is defined with the necessary job parameters specified in the original Server job

      3. Create an orchestrating sequence to invoke the Parallel jobs in the appropriate order

      4. Ensure the sequence has the same job parameters and the original Server job, and that it propagates those parameters to the sequence Job Activity stages appropriately

   4. For each Stage

      1. Add Parallel Stages equivalent to the Server versions

      2. Add appropriate Stage Links, retaining Link ordering where appropriate

      3. Remap Server data types to appropriate Parallel equivalents

      4. Manually configure stage properties to Parallel equivalents, where possible

   5. For Transformer Stages

      1. Convert Server Transformer derivations into Parallel-compatible equivalents

      2. Identify any referenced Custom Routines and replace with a reference to a new Parallel Routine definition

      3. if not already present, create the new Parallel Routine definition for each identified Custom Routine referencing an object file containing a Parallel Routine implementation.

   6. Review and update unit tests

All of these operations are automated by Data Migrators' S2PX.

...