Document toolboxDocument toolbox

Custom Routine Mapping

Owned by John McKeever
Last updated: Jul 20, 2022
4 min read

Expressions, which can incorporate an arbitrarily complex combination of variables, constants, and function calls, appear in a number of places in Server Jobs, but are most commonly used in Transformer Stage Variables and output column Derivations. Whenever S2PX encounters a function call that will not be recognised by the Parallel engine it will attempt to apply some translation to it. S2PX takes a straightforward approach to identifying and translating unknown function calls:

  1. Copy all Server expressions unmodified to their S2PX-generated equivalent Parallel Job(s), on the assumption that an identical Parallel equivalent already exists for it.

  2. Perform text substitution using a set of key/value definitions supplied in the customRoutines mapping section of your S2PX configuration file.

  3. Analyse the resulting expression and adds the typecasting function calls required to handle the differences in Server and Parallel data typing systems.

By design, S2PX does not automatically convert Custom Routine DataSage BASIC to a form compatible with the DataStage Parallel engine, but it does:

Macro Expansion Pre-processing

This process may replace the unknown routine call with a valid Parallel expression.

Users may have Server Jobs using a Custom Server Routine for which there exists a functionally equivalent Parallel Expression. In these cases a user can configure S2PX to automatically replace these unknown Custom Routine references with a Parallel expression. This expression can include any combination built-in Parallel Functions or Custom Parallel Functions which defined in the customRoutines mapping section of your S2PX configuration file. In this regard you can consider the S2PX macro expansion pre-processing step as analogous to macro expansion in C/C++ in that a pre-processor simply substitutes one value for another using a supplied list of key/value pairs. E.g.

MYSERVERFUNCTION({0},{1}): My( Parallel({1}, Expression({0}) ) )

Note that the pre-processor does not validate whether the Parallel expressions you are inserting into your generated Job are valid or not. Ensure that each function in your expansion exists as either…

  • an built-in Parallel DataStage function, or

  • a Custom Parallel DataStage Parallel function, defined in the customRoutines libraries section of your S2PX configuration file.

A future release of S2PX will also consider any Custom Parallel Functions defined in the DSX/ISX files you provide for S2PX analysis/conversion.

Parameter handling

The macro expansion pre-processor also enables you to specify how Custom Routine parameter references should be translated using a simple syntax. When specifying your mappings you don’t need to list the parameters of the Custom Server function, you can just refer to their position using a shorthand syntax of the form {0}, {1}, {2}, etc., where the first parameter is {0}.

Here’s an example:

customRoutines: # A list of mappings from legacy Server functions to Parallel equivalent expressions mapping: # nullToEmptyStr: NullToValue({0}, Str(" ", {1})) # 'nullToEmptyStr(Input.Column, NumSpaces)' is translated to 'NullToValue(Input.Column, Str(" ", NumSpaces))'

Circular references

To prevent circular references macro expansion is NOT recursive, meaning that an expanded expression (on the right of the equals sign) cannot include references to another macro (on the left of the equals sign).

Here’s an example on an invalid configuration:

customRoutines: # A list of mappings from legacy Server functions to Parallel equivalent expressions mapping: # repeatSpaces: Str(" ", {1}) # 'repeatSpaces(NumSpaces)' translates to 'Str(" ", NumSpaces))' nullToEmptyStr: NullToValue({0}, repeatSpaces({1})) # INVALID FUNCTION DEFINITION!

Custom mappings can, however, refer to built-in Parallel functions or Custom Routines, defined in the customRoutines mapping section of your S2PX configuration file.

Expression Validation

Once macros have been expanded, S2PX will parse the resulting expression and identify where typecasting functions need to be inserted. This may present challenges if it encounters a function it doesn’t recognise as it won’t have the information available to determine the function’s input or output datatypes. In this case S2PX makes an assumption that all parameters and return types are strings. This is not a big problem, however, as the resulting Job won’t compile anyway, and addressing the non-existence of the referenced function should be a a user’s first priority!

Those non-existing referenced functions are identified by the Custom Basic Transform Routines Asset Query during your initial S2PX Analysis. The S2PX logging process will also dump log entries which describe

 

© 2015-2024 Data Migrators Pty Ltd.