Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Repeatable Unit Tests require that the job under test produce deterministic data based on a set of predefined inputs and parameters. Calculations based on the current date are common in DataStage jobs but will cause the output produced by jobs to change depending on the date the job executed. This page outlines practices aimed at ensuring jobs using current date calculations are able to be validly tested.

Transformer Stages using CurrentDate(), CurrentTime() or CurrentTimestamp() functions

Unless your date calculation requires an exact date/time a record was being processed, the standard CurrentDate(), CurrentTime() and CurrentTimestamp() functions can be substituted with the DSJobStartDate, DSJobStartTime and DSJobStartTimestamp macros which can be set . This enables you to set them to a specific value during testing. Add DSJobStartDate, DSJobStartTime and/or DSJobStartTimestamp to the parameters clause of the Unit Test Specification and set the appropriate date and time values used during Unit Testing.

Example

Screenshot of Transformer configuration UI showing use of date macro in Derivation.Image Modified

Code Block
languageyaml
...
when:
  parameters:
    DSJobStartDate: 2012-01-15
    DSJobStartTime: 11:05:01
Note

Be careful when setting DSJobStartTimestamp in conjunction with either DSJobStartDate or DSJobStartTime, the MettleCI Unit Testing feature does not enforce that these parameters are logically consistent.

Ignoring columns which are known to be non-deterministic

Rather than ensuring that a job under test produces deterministic data, you may decide to exclude one or more output columns from Unit Test comparisons. This can be done by adding the columns to be ignored to the then clause of the Unit Test Specification.

Example

DataStage Classic Designer Client screenshot showing columns to be ignored in the Unit Test.Image Modified

Code Block
languageyaml
given:
...
when:
...
then:
  - stage: Transform
    link: Output
    path: Transform-Output.csv
    ignore: 
      - CREATION_DATE
      - LAST_UPDATED
Warning

Danger: Ignoring columns will

  • prevent

known,
  • columns containing non-deterministic

behavior
  • from affecting test results

but it is important to remember that ignoring columns is the same as not testing those columns
  • , but also means…

  • those columns are omitted from test comparisons, so unexpected output in those columns, or changes in the output of those columns will not be detected.