There are 2 types of Message Handlers in DataStage.
Project Level Message Handlers are installed in an application directory per Engine and as such will be manually handled and will not be supported by MettleCI.
Job Level Message Handlers are generally not supported by MettleCI but a plugin has been developed to support just their deployment. In other words, Check-in and Deletion from Bitbucket of the Job Level Message Handler files is not supported through MettleCI.
Anti-Pattern
Message Handlers are not an asset of DataStage. See Note below for an explanation why.
"You can use the IBM® InfoSphere® Information Server Manager (the deployment tool) to move, deploy, and control your IBM InfoSphere DataStage® and QualityStage® assets."
However, message handlers cannot be managed individually using the IS Manager or istool command line tool.
In fact, they are an abnormal setting that disguises bad coding issues or job design decisions. The reason they are used is that organizations mandate that Jobs should run without warnings. As a result, the scheduling tools used are often setup to fail jobs if they generate warning messages. Unfortunately, this usually leads to developers becoming stuck in a situation where they don't have enough knowledge/tools/time to resolve the situation.
Organizations should refrain from using Message Handlers and should configure their Job Schedulers to be able to allow Warnings in exceptional circumstances. That way, high standards of development are generally maintained but in the unlikely situation that there is no alternative, Job Warnings can be allowed.
Message Handler Deployment Plug-In
Data Migrators have completed a new Bamboo Plugin to support deployment of Job-Level Message Handlers (dm-dsmsgh-plugin)
The new Plugin includes a task that will read all Message Handlers that follow a "<Job Name>.msh" naming convention and "inject" them into the ISX file of the corresponding Job. The updated ISX file can then be deployed like normal, resulting in both the Job design and Job-Level Message Handler ending up in the target DataStage instance.
The required changes to BIH are as follows:
- All Job-Level Message Handlers in a particular DataStage Project will need to be added to the corresponding Git repository, alongside the existing ISX files. These files need to be named following the "<Job Name>.msh" convention.
- Continuous Integration Bamboo Plans will need to be updated to execute the new "DataStage Message Handlers Task" immediately before the "DataStage Incremental Deployment Task".
- The corresponding <Job Name>.isx jobs need to be checked in to Bitbucket again, in order to fool the Deployment plugin into promoting the Message Handler to Continuous Integration.
- No changes to Bamboo Deployment Plans are required.
How to Find the relevant Local.msh in the Information Server File System.
- Let’s say you have a Job in DataStage called “GR_ACCOUNTS_LOCAL"
- Within the relevant Project filesystem, find the RT_SC<Job Number> directory that corresponds to the Job. The Simplest method is to look in Director.
- Within the RT_SC<Job Number> directory, find the Job-Level Message Handler config file (Local.msh)
- Once found, copy and rename that file to GR_ACCOUNTS_LOCAL.msh and check it into Git alongside the existing ISX file
- Once this file has been checked in, re-check in the corresponding Job. This ensures that automated deployments pick it up.
- The (updated) Continuous Integration Bamboo Plan for the corresponding DataStage Project will be auto-triggered and must complete successfully. This will ensure that subsequent deployments to other environments (Test, QA, Production) will include Job-Level Message Handlers.
Changes to CI Plans
Continuous Integration Bamboo Plans will need to be updated to execute the new "DataStage Message Handlers" Task immediately before the "Incremental DataStage Deployment" Task.
As previously highlighted, the very nature of IBM’s tools for managing Job Level Message Handlers in Information Server means any automation solution for them will be at risk of permanent regression with future IBM Fix Packs and Releases. Even in the absence of such automation challenges, we suggest the general situation with Job Level Message Handlers ought to trigger customers to reconsider their use (or at least modify the tolerance of Warning Messages by dependent systems) and refactor accordingly.