Document toolboxDocument toolbox

DataStage Git Repository Structure

Owned by John McKeever
Last updated: Oct 15, 2020
9 min read

The Git repository created and automatically maintained by MettleCI is structured like this:

. ├── datastage/ # This structure is automatically managed by MettleCI │ ├── Config_One.apt # The apt config files for your project │ ├── Config_Two.apt # │ ├── ... │ ├── DSParams # The DSParams for your project │ ├── Jobs/ # | The structure under the root 'datastage' folder directly reflects │ │ ├── Extract/ # | the structure in root of your DataStage repository. Every time │ │ │ ├── ExtractJob_One.isx # | you check in datastage assets the required folders are created for │ │ │ ├── ExtractJob_Two.isx # | you in your git repository. │ │ │ └── ... │ │ ├── Load/ # │ │ │ ├── LoadJob_One.isx # │ │ │ ├── LoadJob_Two.isx # │ │ │ └── ... │ │ ├── Transform/ # │ │ │ ├── TransformJob_One.isx # │ │ │ ├── TransformJob_Two.isx # │ │ │ └── ... │ │ └── ... │ └── Parameter Sets/ # The 'Parameter Sets' folder contains Parameter Set Value files │ ├── SourceDB1/ # | following the structure used by the DataStage Engine: │ │ └── source1 # | '/datastage/Parameter Sets/<Parameter Set>/<Value File>' │ ├── SourceDB2/ # | │ │ └── source2 # | │ ├── TargetDB/ # | │ │ └── target # | │ └── pGlobal/ # | │ │ └── global # | │ ├── SourceDB1.isx # | The Parameter Set export structure under the root 'datastage' folder │ ├── SourceDB2.isx # | directly relects the structure of your DataStage repository. │ ├── TargetDB.isx # │ ├── pGlobal.isx # ├── filesystem/ # | The content of the file system directory is transferred to the │ ├── deploy.sh # | DataStage engine by MettleCI and deploy.sh is invoked on the DataStage │ ├── datasets/ # | engine to move other directories and files (such as scripts) to the │ └── scripts/ # | appropriate locations for use by the ETL solution │ ├── 10-restore-backups.sql # │ ├── 20-create-test-data.sql # │ └── ... ├── unittest/ # This flat structure is automatically managed by MettleCI │ ├── ExtractJob_One/ # Each job gets its own folder in the unittest folder │ │ ├── dsDataSource1.csv # A test data input file │ │ ├── dsDataSource2.csv # A test data input file │ │ ├── dsDataSource3.csv # A test data input file │ │ ├── dsDataTarget1.csv # A test data output file │ │ ├── ExtractJob_One.yaml # The test specification, associating files to your job's links │ │ ├── ExtractJob_One_OtherTest.yaml # The test specification, associating files to your job's links │ │ ├── ExtractJob_One_AnotherTest.yaml # The test specification, associating files to your job's links │ ├── ExtractJob_Two/ # │ │ ├── {similar to above} # │ ├── LoadJob_One/ # │ │ ├── {similar to above} # │ ├── LoadJob_Two/ # │ │ ├── {similar to above} # │ ├── TransformJob_One/ # │ │ ├── {similar to above} # │ ├── TransformJob_Two/ # │ │ ├── {similar to above} # │ └── ... ├── var.ci # Variable override files provide environment-specific values for ├── var.dev # | each target deployment environment. ├── var.prod # | Search the MettleCI documentation for 'variable override files' ├── var.qa # | These files cover the CI, DEV, PRO, QA, and UAT environments └── var.uat # | as an example

 

 

© 2015-2024 Data Migrators Pty Ltd.