One other challenge in a data integration project as it is with any other IT system is to build a data integration environment that is agile and flexible enough to accommodate the system changes related to business rules. The benefit of building a dynamic system is that the code changes are less frequently done or completely avoided in many cases, so it is very much a de-facto process to look for design opportunities to build a DI system that is dynamic.
A dynamic DI system accommodates variation in the incoming data and is able to respond without system failures or mal-data processing, in many such scenarios the DI environment also gets to be controlled by the Business Team with lesser support from the IT team.
Following are some of the design aspects towards getting a DI system dynamic
- Avoiding hard references, usage of parameter variables
- Usage of lookup tables for code conversion
- Setting and managing threshold value through tables
- Segregating data processing logics into common reusable components
- Ensuring that the required processes are controllable by the Business team with the required checks built in
Avoiding hard references, usage of parameter variables:
In scenarios like defining simple database connections or to using a “Business Unit Number” for filtering data while extracting required data from a source system can be parameterized through variables. In the case of database parameterization it enables easier code movement across development-test-production environments and in the scenario of the Business Unit Number parameterization it enables running the same DI program for one other Business Unit with same data by just changing the parameter variable of Business unit Number
In scenarios like defining simple database connections or to using a “Business Unit Number” for filtering data while extracting required data from a source system can be parameterized through variables. In the case of database parameterization it enables easier code movement across development-test-production environments and in the scenario of the Business Unit Number parameterization it enables running the same DI program for one other Business Unit with same data by just changing the parameter variable of Business unit Number
Usage of lookup tables for code conversion
In scenarios like where the incoming data value has to be converted to one other “standard value”, we usually write a IF..THEN… ELSE… syntax, here again we can bring in dynamism by having a lookup table which would carry the incoming value as one column and the standard value to be replaced as another column. The benefit is if there is any further change to the standard value the lookup table can be updated without opening the code and as well we can insert a new record into the lookup in case we need get the DI process handle a code conversion for a new incoming value.
In scenarios like where the incoming data value has to be converted to one other “standard value”, we usually write a IF..THEN… ELSE… syntax, here again we can bring in dynamism by having a lookup table which would carry the incoming value as one column and the standard value to be replaced as another column. The benefit is if there is any further change to the standard value the lookup table can be updated without opening the code and as well we can insert a new record into the lookup in case we need get the DI process handle a code conversion for a new incoming value.
We shall see more details and other design aspects in the coming days.
You might want to read these awesome related posts Data Integration Challenge
0 comments:
Post a Comment