How the Parallel Unit Test Harness Integrates with DataStage

DataStage Parallel jobs are compiled into an osh script which describes the processes (operators) required to implement the job to the desired degree of parallelism. When DataStage begins executing a Parallel job it forwards this osh script to the osh command line program which determines the required structure of conductor, section leader, and player processes, before then invoking these processes to execute your job.

The MettleCI Unit Testing Harness intervenes in the invocation of the osh command and inspects the value of the $DM_ENABLE_UNIT_TESTING parameter to determine if the job invocation is a normal execution, Interception, or Unit Test execution. Normal execution requests are forwarded to the regular IBM-supplied osh command. Requests for Interception or Unit Test executions, however, will cause MettleCI to modify your job’s osh script before forwarding it to the osh command.

MettleCI achieves this by modifying your osh execution infrastructure during installation of the Unit Test harness. Specifically, the installation process does the following:

renames the osh command to osh-ibm (to identify it as the original IBM-supplied osh command)
installs a new executable called osh-dm (to identify it as a new command supplied by Data Migrators)
replaces the original osh command with a new symbolic link named osh which points to osh-dm

Contrast an unmodified ../PXEngine/bin directory:

>$ pwd
/opt/IBM/InformationServer/Server/PXEngine/bin
>$ ls -al osh*
-rwxrwxr-x. 1 dsadm dstage 73224 Mar 23 2019 osh
-rwxrwxr-x. 1 dsadm dstage 10648 Mar 23 2019 oshprinthost
-rwxrwxr-x. 1 dsadm dstage 19192 Mar 23 2019 oshwrapper
>$

…with one in which the MettleCI Unit Test Harness has been installed:

$> pwd
/opt/IBM/InformationServer/Server/PXEngine/bin
$> ls -al osh*
lrwxrwxrwx 1 dsadm dstage 6 Jan 21 18:50 osh -> osh-dm
-rwxr-xr-x 1 dsadm dstage 2668 Jan 21 18:20 osh-dm
-rwxr-xr-x 1 dsadm dstage 69048 Jul 5 2018 osh-ibm
-rwxrwxr-x. 1 dsadm dstage 10648 May 10 2017 oshprinthost
-rwxrwxr-x. 1 dsadm dstage 19192 May 10 2017 oshwrapper
$>

To execute your jobs your DataStage client will call the osh symbolic link. This will invoke the MettleCI executable osh-dm which will identify any requirement to modify the supplied osh script before invoking osh-ibm with that script as the input. The MettleCI Unit Test Harness can be temporarily (or permanently) disabled by modifying the osh symlink to point directly to the IBM-supplied original:

$> ln -fs osh-ibm osh

If you apply a DataStage fix pack that contains a new version of osh, you must make provisions to restore the symbolic link which invokes osh-dmotherwise the patch may replace the symlink with a new version of osh. As discussed in https://datamigrators.atlassian.net/wiki/spaces/MCIDOC/pages/1457750170, the easiest way to do this is run the harness uninstaller prior to applying the patch. Harness installation is reversible and uninstallation will restore the directory structure to the non-symbolically linked version. This uninstall process performs the same set of checks as the install process to ensure it doesn't do anything destructive.

MettleCI Documentation

How the Parallel Unit Test Harness Integrates with DataStage

See also