Document toolboxDocument toolbox

Deploying MettleCI repository templates to your Git platform

Owned by John McKeever
Last updated: Jun 11, 2024
6 min read

A Git repository (or ‘repo’) is a local filesystem directory or remote storage space where your project files and their revision history are stored.

Git repositories can be local (hosted locally on your computer) or remote (hosted remotely on services like GitHub, Azure DevOps, or Atlassian Bitbucket) allowing for collaboration among multiple developers regardless of their physical location. Note that a ‘remote' repository can also include repositories residing on a self-hosted versions of a Git platform resident within your organisation.

MettleCI ships with a zip file containing a two local Git repositories:

  • A compliance rules repository template with a name of the form dm-compliance-rules-NNN.zip

  • A DataStage asset repository template with a name of the form mettleci-repo-template-NNN.zip

… where NNN represents the version of the repository you have downloaded.

Compliance Rules Repository Template

This local repository provides a set of template MettleCI Compliance rules. It’s possible that your organisation may be content to adapt the supplied rules to fit your needs and leave them in a local Git repository on your engine. If this is the case, then see our guidance on using this local Git repository for Compliance checking.

Alternatively, your team may wish to promote the local repository to a remote repository where developers can collaborate on extending augmenting the rules as necessary. To do this you will need to deploy your compliance repository to a remote host. Start this by creating an empty remote Git repository in your Git system of choice and copy its reference URL. Here are some examples of remote repositories on some popular Git platforms …

GitHub

Atlassian Bitbucket

Azure DevOps

GitHub

Atlassian Bitbucket

Azure DevOps

image-20240419-023009.png
image-20240419-023131.png

Next, open a shell on a system where you will unzip the and follow these steps:

Note that this example uses a trunk branch called main whereas some repositories/organizations will use the word master.

# ---------------------------------------- # 1 # Verify you have a functional Git command line installed # Go to your local compliance repository directory # ---------------------------------------- $> git --version git version 2.43.0 # ---------------------------------------- # 2 # Unzip the supplied compliance rules file # ---------------------------------------- $> unzip dm-compliance-rules-278.zip Archive: dm-compliance-rules-278.zip creating: compliance/ inflating: compliance/Password Param Type not Encrypted.pjb.grm <snip> inflating: compliance/Default Naming.qjb.grm # inspect your repository files $> ls -l total 936 drwxr-x---@ 134 johnmckeever staff 4.2K 10 Nov 17:05 compliance -rw-r--r--@ 1 johnmckeever staff 354K 12 Apr 16:58 dm-compliance-rules-278.zip -rw-r--r--@ 1 johnmckeever staff 111K 12 Apr 16:58 mettleci-repo-template-73.zip $> cd compliance $> ls -l total 1560 -rw-r-----@ 1 johnmckeever staff 1.7K 10 Nov 17:05 Adjacent Transformers.pjb.grm <snip> -rw-r-----@ 1 johnmckeever staff 13B 10 Nov 17:05 version.txt # ---------------------------------------- # 3 # Re-define your local repository as a clone of your newly-create remote repository # using the remote reference to your empty Compliance repository provided by your Git system # ---------------------------------------- $> git remote add origin https://yourusername@bitbucket.org/yourusername/our-compliance-repo.git # Note that if you need to subsequently re-point your repository to a different remote repository for # any reason (e.g. you make a mistake, or your situation changes) you can use the 'set-url' command. # This is only provided here by way of example. You shouldn't need to run this under normal circumstances. # # $> git remote set-url origin https://yourusername@bitbucket.org/yourusername/our-compliance-repo.git # ---------------------------------------- # 4 # Add all the Compliance rules shipped in the Data Migrators-supplied Zip file to the staging area # ---------------------------------------- $> git add --all . # ---------------------------------------- # 5 # Commit the Compliance rules you just added # Use any commit message you like # ---------------------------------------- $> git commit -am "initial commit to remote, default rules" [main (root-commit) 2b95553] default rules Committer: myusername <user@engine> 24 files changed, 1421 insertions(+) create mode 100644 Adjacent Transformers.pjb.grm create mode 100644 CCMigrateTool Stages.pjb.grm create mode 100644 CCMigrateTool Stages.sjb.grm create mode 100644 Database Row Limit.pjb.grm <snip> # ---------------------------------------- # 6 # Push those commits to the new remote repository # ---------------------------------------- $> git push -u origin main Password for 'https://yourusername@bitbucket.org': Counting objects: 24, done. Delta compression using up to 2 threads. Compressing objects: 100% (23/23), done. Writing objects: 100% (24/24), 20.74 KiB | 0 bytes/s, done. Total 24 (delta 1), reused 0 (delta 0) To https://yourusername@bitbucket.org/yourusername/our-compliance-repo.git * [new branch] main -> main Branch main set up to track remote branch main from origin. # ---------------------------------------- # 7 # Verify the status of your local repository # ---------------------------------------- $> git show-ref 2b95553dc7572c58e12be36c4f59a52eaf61af8b refs/heads/main 2b95553dc7572c58e12be36c4f59a52eaf61af8b refs/remotes/origin/main # Done!

DataStage Asset Repository Template

The initial process for deploying the DataStage Asset Repository Template is identical to that of the Compliance Repository Template. Start by creating an empty repository to host your DataStage assets and build pipeline definitions. Copy a reference to this repository then follow the guide below.

Note that the DataStage Asset Repository Template contains example pipelines for a number of build systems. You should delete those which are irrelevant to your organisation before committing it to your remote repository for the first time.

# ---------------------------------------- # 1 # Verify you have a functional Git command line installed # Go to your local compliance repository directory # ---------------------------------------- $> git --version git version 2.43.0 # ---------------------------------------- # 2 # Unzip the supplied repository template file # ---------------------------------------- $> unzip mettleci-repo-template-73.zip Archive: mettleci-repo-template-73.zip creating: mettleci-repo-template/ inflating: mettleci-repo-template/var.prod creating: mettleci-repo-template/filesystem/ <SNIP> # Inspect your repository files $> cd mettleci-repo-template $> ls -l total 64 -rw-r--r-- 1 johnmckeever staff 1.0K 17 Apr 14:16 README.md drwxr-xr-x 5 johnmckeever staff 160B 17 Apr 14:19 bamboo-specs drwxr-xr-x 7 johnmckeever staff 224B 19 Mar 13:41 datastage drwxr-xr-x 4 johnmckeever staff 128B 19 Mar 13:41 filesystem drwxr-xr-x 7 johnmckeever staff 224B 19 Mar 13:41 pipelines <SNIP> # Verify that this directory is indeeed a local Git repository $> git status On branch main No commits yet Untracked files: (use "git add <file>..." to include in what will be committed) .gitattributes README.md bamboo-specs/ datastage/ <SNIP> # ---------------------------------------- # 3 # Customise your repository to your needs # This example removes some pipline examples residing under the 'pipelines' directory # ---------------------------------------- $> rm -rf ./pipelines/system-not-required/ $> rm -rf ./pipelines/other-system-not-required/ # ---------------------------------------- # 4 # Re-define your local repository as a clone of your newly-create remote repository # using the remote reference to your empty DataStage Asset repository provided by your Git system # ---------------------------------------- $> git remote add origin https://yourusername@bitbucket.org/yourusername/our-datastage-repo.git # ---------------------------------------- # 5 # Add the DataStage Asset template shipped in the Data Migrators-supplied Zip file to the staging area # ---------------------------------------- $> git add * # ---------------------------------------- # 6 # Commit the DataStage Asset repository template you just added # Use any commit message you like # ---------------------------------------- # Create the commit $> git commit -am "Initial commit of repository template to remote" [master (root-commit) 3202f35] initial commit to remote, repository template 55 files changed, 7028 insertions(+) create mode 100644 README.md create mode 100644 datastage/Parameter Sets/.placeholder <snip> # ---------------------------------------- # 7 # Push those commits to the new remote repository # ---------------------------------------- $> git push -u origin main Counting objects: 24, done. Delta compression using up to 2 threads. Compressing objects: 100% (23/23), done. Writing objects: 100% (24/24), 20.74 KiB | 0 bytes/s, done. Total 24 (delta 1), reused 0 (delta 0) To https://yourusername@bitbucket.org/yourusername/our-compliance-repo.git * [new branch] main -> main Branch main set up to track remote branch main from origin. # ---------------------------------------- # 8 # Verify the status of your local repository # ---------------------------------------- $> git --no-pager log --decorate=short --pretty=oneline -n1 18f17799b49845d503b1ad468c35e9acc64462b5 (HEAD -> master, origin/main, origin/HEAD) Initial commit of repository template to remote # Done!

 

© 2015-2024 Data Migrators Pty Ltd.