MettleCI works with assets that are kept in Git repositories, notably the repository associated with a DataStage project and the repository associated with the compliance rules.
...
Achieving a usable MettleCI instance includes the configuration of these repositories within GitLab.
...
Note: GitLab differs from similar tools in its application of meanings to common terms.
| Info |
|---|
Avoiding confusion over common terms in GitHub, Bitbucket, and GitLab Despite Git being a well established standard, different vendors' Git-based version control systems can use terminology like Repository and Project differently, creating the potential for confusion. In other platforms such as GitHub and Bitbucket, “repositories” contain the Git code repository along with other project-related assets such as issues, contribution metrics, etc. However |
...
the GitHub community often uses the terms repository and project interchangeably. In GitLab, the overarching container is called a Project. A Project includes the Git repository, issues, merge requests, milestones, and |
...
more. It's important to make this distinction because you import |
...
a Project in GitLab, regardless of whether that is called |
...
a Repository elsewhere. In GitLab, the |
...
Repository is a component part of a Project. Thus, GitLab tightly couples project and repository, and each project |
...
may have only one Git repository. |
Note: One may keep MettleCI Compliance rules in the same repository as the DataStage assets but it’s generally recommended that a repository be dedicated to those Compliance rules intended for use by multiple pipelines (across all your DataStage-related GitLab projects). That means GitLab pipelines must be configured to explicitly clone that Compliance repository so it’s available when the pipeline reaches the Compliance execution step. The example GitLab pipeline provided with MettleCI reflects this dependency.
Because the details of how to create and manage projects vary among GitLab releases, we speak in general terms only, and note MettleCI specific considerations.
Steps
Create a blank GitLab project by selecting new project from your GitLab dashboard, or visiting http://your.gitlab.server/projects/new. The recommended practice is to create a new GitLab project for each DataStage project, plus one for your Compliance rules. You may want to collect your GitLab projects into groups.
...
See https://
...
...
...
user/project/ for more information on projects.
Clone the empty repository to somewhere convenient, such as your local DataStage client machine. Create a README file in the root folder of the repository which ensures the repository is initialized, has a default branch, and can be cloned
...
. A sample README file is supplied in the template repository supplied with MettleCI.
For integration with MettleCI Workbench, with CI/CD pipelines, and with external entities (such as code authoring systems) in general, you will need to set up access to the repository associated with the project.
Decide what credentials type you will use: userid/password, userid/PAT, deploy keys (a kind of SSH key) or regular SSH. SSH is recommended and this page has more details on why: Configuring MettleCI Workbench SSH Authentication to GitLab
If following the above recommendation, create a public/private key pair and then add it to the project. SeeAdding public keys to GitLab for more details.
Set up access between GitLab and the MettleCI Workbench service (typically running on your DataStage Engine host)
Obtain the “clone URL” for the project
Configure the project in MettleCI Workbench. Refer to MettleCI Workbench Configure First Project for instructions.
Test this setup by using MettleCI Workbench to run a Compliance check (confirming read-only access to Compliance) and perform a commit to the repository (confirming write access to the Project repository).
Set up access between GitLab and your build server. The easiest way to do this is to create a (project-specific or public) deploy key.
...
Public keys can be used across multiple GitLab projects,
...
once the project administrator grants access, while private keys only grant access to one project.
...
= = = = = = =
...
Repository - wwi_gitlab_ds117_ds117 (same as project)
Pipeline - wwi_gitlab_ds117
Hazy if this naming works out, we will see
This page in admin area is how to create new project
http://test1-gitlab.datamigrators.io/projects/new
Project is here http://test1-gitlab.datamigrators.io/lar/wwi_gitlab_ds117_ds117
...
also can use this cmd from (somewhere)?
git push --set-upstream git@ec2-3-94-105-174.compute-1.amazonaws.com:lar/$(git rev-parse --show-toplevel | xargs basename).git $(git rev-parse --abbrev-ref HEAD)
after creating it, clone it on a machine where you can seed it
git@ec2-3-94-105-174.compute-1.amazonaws.com:lar/wwi_gitlab_ds117_ds117.git or http://ec2-3-94-105-174.compute-1.amazonaws.com/lar/wwi_gitlab_ds117_ds117.git
I need to do this setup
| Code Block |
|---|
git config --global user.name "Larry Pieniazek"
git config --global user.email "larry.pieniazek@datamigrators.com"
|
Create a new repository
| Code Block |
|---|
git clone http://ec2-3-94-105-174.compute-1.amazonaws.com/lar/wwi_gitlab_ds117_ds117.git
cd wwi_gitlab_ds117_ds117
touch README.md
git add README.md
git commit -m "add README"
git push -u origin master |
May need to create and register an ssh key
C:\data\DM_temp\git\test1-gitlab>ssh-keygen -t rsa -b 4096 -C "lar@test1-gitlab.datamigrators.io"
Generating public/private rsa key pair.
Enter file in which to save the key (C:\Users\lpieniaz/.ssh/id_rsa): C:\data\DM_temp\git\test1-gitlab\id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\data\DM_temp\git\test1-gitlab\id_rsa.
Your public key has been saved in C:\data\DM_temp\git\test1-gitlab\id_rsa.pub.
The key fingerprint is:
SHA256:X5rl0YnMs/w9zUzcm1KyGLv957gI4C7dJgtMwj0T4YY lar@test1-gitlab.datamigrators.io
The key's randomart image is:
+---[RSA 4096]----+
| . |
| o . |
| E + |
| . o . o o . |
| o = S O o..|
| + + o O = .+|
| o...= B +++|
| .oo o+.= =|
| .o+ .o.+=|
+----[SHA256]-----+
C:\data\DM_temp\git\test1-gitlab>dir
Volume in drive C is Windows
Volume Serial Number is 6CA6-6484
Directory of C:\data\DM_temp\git\test1-gitlab
01/12/2021 04:02 PM <DIR> .
01/12/2021 04:02 PM <DIR> ..
01/12/2021 04:02 PM 3,247 id_rsa
01/12/2021 04:02 PM 760 id_rsa.pub
2 File(s) 4,007 bytes
2 Dir(s) 50,928,996,352 bytes free
C:\data\DM_temp\git\test1-gitlab>mkdir .ssh
C:\data\DM_temp\git\test1-gitlab>move id* .ssh
C:\data\DM_temp\git\test1-gitlab\id_rsa
C:\data\DM_temp\git\test1-gitlab\id_rsa.pub
2 file(s) moved.
C:\data\DM_temp\git\test1-gitlab>dir
Volume in drive C is Windows
Volume Serial Number is 6CA6-6484
Directory of C:\data\DM_temp\git\test1-gitlab
01/12/2021 04:02 PM <DIR> .
01/12/2021 04:02 PM <DIR> ..
01/12/2021 04:02 PM <DIR> .ssh
0 File(s) 0 bytes
3 Dir(s) 50,928,996,352 bytes free
C:\data\DM_temp\git\test1-gitlab>cd .ssh
C:\data\DM_temp\git\test1-gitlab.ssh>dir
Volume in drive C is Windows
Volume Serial Number is 6CA6-6484
Directory of C:\data\DM_temp\git\test1-gitlab.ssh
01/12/2021 04:02 PM <DIR> .
01/12/2021 04:02 PM <DIR> ..
01/12/2021 04:02 PM 3,247 id_rsa
01/12/2021 04:02 PM 760 id_rsa.pub
2 File(s) 4,007 bytes
2 Dir(s) 50,929,061,888 bytes free
C:\data\DM_temp\git\test1-gitlab.ssh>
...
11/26/2020 07:13 PM <DIR> .
11/26/2020 07:13 PM <DIR> ..
11/26/2020 07:13 PM 1,675 id_rsa
11/26/2020 07:13 PM 416 id_rsa.pub
10/19/2020 04:39 PM 604 known_hosts
3 File(s) 2,695 bytes
2 Dir(s) 50,917,089,280 bytes free
C:\Users\lpieniaz.ssh>
Sorted this out after some trouble.
installed it here
http://test1-gitlab.datamigrators.io/profile/keys
...
Then used the template to set up the repo files along with my project’s files
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117>git status
On branch master
Your branch is up to date with 'origin/master'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
.gitattributes
.gitlab-ci.yml
compliance/
datastage/
filesystem/
unittest/
var.ci
var.develop
var.hf_ci
var.hf_develop
var.perf
var.prod
var.qa
nothing added to commit but untracked files present (use "git add" to track)
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117>git add *
warning: CRLF will be replaced by LF in compliance/WORKBENCH/Adjacent Transformers.pjb.grm.
The file will have its original line endings in your working directory
…
warning: CRLF will be replaced by LF in var.qa.
The file will have its original line endings in your working directory
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117>
then commit and push
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117>git commit -m "seed the repository with GitLab template and WWI project files
[master 982d4ab] seed the repository with GitLab template and WWI project files
147 files changed, 117738 insertions(+)
create mode 100644 .gitattributes
create mode 100644 .gitlab-ci.yml
create mode 100644 compliance/.DS_Store
create mode 100644 compliance/WORKBENCH/Adjacent Transformers.pjb.grm
…
create mode 100644 var.qa
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117>
C:\data\DM_temp\git\test1-gitlab\wwi_gitlab_ds117_ds117>git push
Enumerating objects: 161, done.
Counting objects: 100% (161/161), done.
Delta compression using up to 8 threads
Compressing objects: 100% (155/155), done.
Writing objects: 100% (160/160), 2.21 MiB | 1.93 MiB/s, done.
Total 160 (delta 17), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (17/17), done.
To test1-gitlab.datamigrators.io:lar/wwi_gitlab_ds117_ds117.git
569988e..982d4ab master -> master
...
You can also use a deploy token instead.
This page, while targeted at MettleCI Workbench setup, also covers how to create Deploy keys (Configuring MettleCI Workbench SSH Authentication to GitLab ). They are created the same way you create other SSH keys, and are actually a pair of keys, public and private.
Once you have a deploy key you will place the public part in the http://your.gitlab.server/your_project/-/settings/repository repository portion of the project settings page.
Deploy tokens are generated for you by GitLab via the same project repository settings page and you must record them for later use. You only get a single opportunity to view them during the setup process. This makes them both more secure and less convenient than deploy keys, which are retained on the file system of the machine communicating with GitLab. See GitLab deploy tokens for more information.