Data Integration Challenge – Understanding Lookup Process –II

Most of the leading products like Informatica, DataStage support all the three ways of lookup process in their product architecture. The following table lists ‘when to use’ and ‘when not to use’ the particular type of lookup process.

Lookup When To Use When Not To Use Direct Query (Uncached lookup in Informatica) When the lookup process is to be invoked only once or a very few times The ETL server and the database are co-located or well connected Reading in large volume of source records and executing lookup queries for every incoming record can be costly in terms network load, query parsing, data parsing and disk hits Querying the same set of records again and again Join Query (Joiner Transformation or a Join on the Source Qualifier in Informatica) When multiple records are returned by the lookup process and all the returned records are required for further processing Both the source and lookup table are on the same database When the source record performs a lookup based on some other ‘TRUE’ condition i.e., not all the records that are read from source do a lookup When the source and lookup table columns are not indexed by the ‘lookup condition’ When the database memory is fully utilized and the Outer Joins are badly executed by the database Cached Query (Cached Lookup in Informatica or Hash files in Datastage) Many times the lookup process being executed Presence of Large volume of data in the looked up table Set of records from the lookup table used by multiple jobs Disk space is a constraint Multiple records from lookup required for processing

Advantage Cache Lookup:

The advantages of using cache file based lookups are that

• Fields that are present in the cache file is only that is needed by the lookup process so when querying the cache file the return would be faster as compared to the lookup table that might have more fields present
• The data structure of the cache file would be designed in such that the query from the ETL server is easily understood without any additional layer like SQL

Though in general it is said in user manuals that usage of cache files is best suited for low volume of lookup but in practical scenarios I have seen cache files are more valuable in terms of performance when the lookup records are huge.

Dynamic Cache: We have the concept of Dynamic Cache in Informatica and as well in Hash files of Datastage where you can Insert/Update or delete records from these cache file. The feature of  updating the cache files is useful when we want to keep the cache file and the lookup table in sync.

Handling Multiple Return Records: Handling the return of multiple records by a lookup process is still a challenge not implemented in any of the leading products – limited to my knowledge. Probably in release 9 Informatica’s lookup can have a parameter for defining the number of records to return as an array like in its Normalizer transformer.

In Part III we shall see some of the things to be considered for better performance when using the lookup process

You might want to read these awesome related posts Data Integration Challenge