Document toolboxDocument toolbox

Link Partitioner Stage

Summary

The Link Partitioner stage is an active stage which takes one input and allows you to distribute partitioned rows to up to 64 output links. The stage expects the output links to use the same metadata as the input link.

IBM documentation

Server Stage: Link Partitioner Stages - IBM Documentation

Parallel Stages: Local containers - IBM Documentation

Conversion Notes

Structural changes

The output of the Server Link Partitioner translation process follows the same structure as the Server Link Partitioner stage itself: a single inputs with multiple outputs. The Server Link Partitioner stage is replaced with an identically-named Parallel Local Container within which is the following implementation:

  • Each Server Link Partitioner input link is translated into a container input link.

  • Each Server Link Partitioner output link is translated into a container output link.

  • The container input link has a Parallel Transformer Stage which acts to rename columns to align with expected output metadata definitions.

    • The outputs of this transformer has constraint definitions which replicate the behaviour from the original partitioner, in most cases the exact partition algorithm is replicated the exceptions are Hash and Random partition algorithms.

  • In the case of hash partitioning a checksum stage is also inserted for the hashing component before the transformer. This is because there are no hashing functions directly available in Px transformers like CRC32 in Server.

See Parallel Job Structural Differences.

Hash Partition Case:

 

Server features not supported

Feature

Asset Query (?)

Comment

Feature

Asset Query (?)

Comment

Random Partitioning Algorithm

TBC

Random partitioning is supported but the underlying random function is technically different. This is unlikely to matter since the previous and current random functions are non-deterministic anyway.

Hash Partitioning Algorithm

TBC

Hash partitioning is supported but the underlying algorithm will be different. This means that the number of records sent to different output links and their order will be different.



 

© 2015-2024 Data Migrators Pty Ltd.