While most Most DataStage jobs can be tested by using MettleCI via MettleCI’s Unit Testing function simply to replace by replacing input and output stages. However, some job designs - while commonplace - will necessitate a more advanced Unit Testing configuration. This page outlines MettleCI Unit Testing Test Spec patterns that can be applied to test best match these job designs.
| Table of Contents |
|---|
Input stage with rejects
The Input stage can be Unit Tested by including both read and reject links in the given clause of the Unit Test Spec. MettleCI Workbench will automatically detect this Unit Testing pattern during creation of new Unit Tests.
The CSV data specified for the rejects link should contain records used for Unit Testing that will actually test the flow of records through the reject path(s) of the job.
| Info |
|---|
If you use MettleCI’s automated Unit Test creation function for a Job that needs this Unit Testing pattern, MettleCI will ensure the resulting Spec in that new Unit Test reflects the above approach. |
| Code Block | ||
|---|---|---|
| ||
given:
- stage: Input
link: Read
path: Input-Read.csv
- stage: Input
link: Rejects
path: Input-Rejects.csv
when:
... |
Output stage with rejects
The output stage can be Unit Tested by including
the write link in the then clause of the Unit Test Spec; and
the reject in the given clause of the Unit Test Spec
.
The CSV data specified for the rejects link should contain records used for Unit Testing that will actually test the flow of records through the reject path(s) of the job.
| Info |
|---|
If you use MettleCI’s automated Unit Test creation function for a Job that needs this Unit Testing pattern, MettleCI will ensure the resulting Spec in that new Unit Test reflects the above approach. |
| Code Block | ||
|---|---|---|
| ||
given:
- stage: Output
link: Rejects
path: Output-Rejects.csv
when:
...
then:
- stage: Output
link: Write
path: Output-Write.csv
|
Stored Procedure stage
A Stored Procedure Stage will not only connect to an external Database for processing but it will also produce output records which are not deterministic. To MettleCI’s Unit Test this job design, function needs to be made aware of the Stored Procedure Stage needs to be removed from the job and replaced with Unit Test Datareplaced by unit test data during unit test execution. This is done by adding the input link in to the then clause of the Unit Test Spec and the output link in the given clause of the Unit Test Spec.
The CSV data input specified by the given clause contains the data that will test become the flow of records from the Stored Procedure stage. The data could simulate what would be produced by the real stored procedure if it had processed the Unit Test input records, however this is not a requirementthey don’t have to.
| Code Block | ||
|---|---|---|
| ||
given:
- stage: StoredProcedure
link: Output
path: StoredProcedure-Output.csv
when:
...
then:
- stage: StoredProcedure
link: Input
path: StoredProcedure-Input.csv |
Classic Surrogate Key Generator stage
The classic Surrogate Key Generator stage will generate sequential keys from a given start value which is (typically set via a Job Parameter). To ensure that the same surrogate key values are generated during unit test execution will always match the expected valuesvalues generated by the Surrogate Key Generator stage remain the same between each execution of Unit Testing, add a fixed value for the start value Job Parameter in the when clause of the Unit Test Spec.
| Code Block | ||
|---|---|---|
| ||
given:
...
when:
job: KeyGeneratorExample
parameters:
START_KEY: 100
then:
... |
Database or Flat File-backed Surrogate Key Generator stage
Surrogate Key Generators backed by a Database or a Flat File will produce output records which are not deterministic. The use of a Database-backed Surrogate Key Generator will also require a live connection to an external Database which is not ideal for Unit Testing. To Unit Test job designs containing this type of Surrogate Key Generator, the Surrogate Key Generator stage needs to be removed from the job and replaced with Unit Test Data. This is done by adding the input link in the then clause of the Unit Test Spec and the output link in the given clause of the Unit Test Spec.
The CSV data input specified by the given clause contains the data that will test become the flow of records from the Surrogate Key Generator stage. The data could simulate what would be produced by the real Surrogate Key Generator if it had processed the Unit Test input records, however this is not a requirementit doesn’t have to. The easiest way to simulate the Surrogate Key Generator output records using MettleCI Workbench would be to copy the CSV specified in the then clause, add a new column to represent the generated key and set appropriate key values.
| Code Block | ||
|---|---|---|
| ||
given:
- stage: KeyGenerator
link: Output
path: KeyGenerator-Output.csv
when:
...
then:
- stage: KeyGenerator
link: Input
path: KeyGenerator-Input.csv |
Sparse Lookup stage
When building DataStage jobs using the Lookup stage, performing a Sparse or Normal lookup is as simple as changing the lookup type of the reference Database stage. When However, when a DataStage job is compiled to OSH and executed, the lookup Lookup stage is not used to perform the sparse lookup. Instead, the Lookup is replaced with the Database operator which is responsible for reading input rows, looking up values from the database and producing output records. It is for this reason that all Database log messages in the DataStage Director are attributed to the lookup stage and why the Database stage never appears in the Monitor of the DataStage Director.
It is not possible for the MettleCI Unit Test Harness feature to change the lookup from Sparse to Normal without fundamentally transforming the run-time job design, this . Doing so would invalidate any Unit Test results, defeating the purpose of this MettleCI function. To Unit Test job designs using sparse Sparse lookups, add the input link in the then clause of the Unit Test Spec and the output link in the given clause of the Unit Test Spec.
The CSV data input specified by the given clause contains the data that will test become the flow of records from the Sparse Lookup stage. The data could simulate what would be produced by the real Sparse Lookup Stage if it had processed the Unit Test input records, however this is not a requirementthey don't have to.
| Code Block |
|---|
given:
- stage: SparseLookup
link: Output
path: SparseLookup-Output.csv
when:
...
then:
- stage: SparseLookup
link: Input
path: SparseLookup-Input.csv |
Alternative approach for testing Sparse-Lookup-heavy jobs
For jobs where the vast majority of job logic is implemented using Sparse Lookup stages, replacing all lookups with Unit Test data will would result in little-to-no DataStage logic being tested (as illustrated below).
For this type of Job design, an alternative testing approach is to leave the sparse lookup Sparse Lookup in place and replace the input and output stages with Unit Test data. A live Database connection will be required during testing but the when clause can be used to set job parameters that dictate database connection settings.
| Note |
|---|
Technically this is an Integration Test, not a Unit Test |
: The Unit Test Harness does not provide any functionality for populating database reference tables with Unit Test data prior to test execution, users are responsible for managing Integration Test setup and tear down through governance and/or CI/CD pipeline customisation. |
| Code Block | ||
|---|---|---|
| ||
given:
- stage: Source
link: Input
path: Source-Output.csv
when:
parameters:
DbName: MyUnitTestDb
DbUser: MyUnitTestUser
DbPass: {iisenc}dKANDae233DJIQeiidm==
then:
- stage: Taret
link: Output
path: Target-Output.csv |
