Do you have a vision for your Business Intelligence?

A quick search on the internet on Business Intelligence brings up a myriad of results:

a)  Big Data analytics helps to analyze large volumes of data – Is this BI?

b)  Data Virtualization brings together enterprise data from multiple, disparate sources– Is this BI?

c)   Data discovery tools offer agility and high performance for faster data exploration – is this BI?

None of this is Business Intelligence. The term Business Intelligence (BI) is often confused for a technology. However, the technologies that go by the name of BI are only a means to an end. The end still remains the information, insights and actions delivered through this technology platform. This understanding is critical to the success of Business Intelligence.

An often cited metric about Business Intelligence is that more than 50% of BI projects fail. Even as one questions the veracity of the claim, it is but common knowledge that BI projects drag on for a long period without producing concrete business deliverables. One of the primary reasons that this happens and that BI projects are not considered successful is that success in a BI project is seldom established.

It is not often that organizations make a conscious effort to create a business intelligence vision, which underlines the success criteria for the BI platform. In many organizations, BI initiatives start with one section of the business and gradually evolve to cover other areas. When this happens or when there is an attempt to consolidate several independent local BI initiatives together into an organization-wide BI platform without a clear vision and a direction, the outcome is often less than desirable.

The BI vision is essentially a set of clearly defined business objectives and a bunch of relevant metrics to track the progress towards the objectives and the final outcome. It is important to ensure that there are metrics that reflect both internal (mostly relevant at the divisional level) and external effectiveness (relevant at the organization / market level) and that they are aligned with each other and to the business objectives.

There are several industry-standard methodologies such as the Balanced Scorecard, the Performance Pyramid that are available to deduce strategic metrics or performance indicators in alignment to the overall strategy / business objectives.

Do you want your BI program to be success? Do you have a vision for your Business Intelligence?

Linking BI Technology, Process and People – A Theory of Constraints (TOC) View

With the advent of a new year, let me do a recap of what I have discussed through my 15 odd posts in 2007 and also set the direction for my thoughts in 2008.

I started with the concept (http://blogs.hexaware.com/business_intelligence/2007/06/business-intell.html) of BI Utopia in which information is available to all stakeholders at the right time, right time and in the right format. The bottomline is to help organizations compete on analytics in the marketplace. With that concept as the starting point, I explored some technology enablers like Real Time Data Integration, Data Modeling, Service Oriented architecture etc. and also some implementation enablers like Agile Framework, Calibration of DW systems and Function points based estimation for DW. In my post on Data Governance, I introduced the position of a CDO (Chief Data Officer) to drive home the point that nothing is possible (atleast in BI) without people!

To me, BI is about 3 things – Technology, Process, People. I consider these three as the holy triumvirate for successful implementation of Business Intelligence in any organization – Not only are the individual areas important by itself but the most important thing is the link between these 3 areas. Organizations that are serious about ‘Analytics’ should continuously elevate their technology, process & people capability and more importantly strengthen the link between them – afterall, any business endeavor is only as good as its weakest link.

Theory of Constraints (http://en.wikipedia.org/wiki/Theory_of_Constraints) does offer a perspective, which I feel is really useful for BI practitioners. I will explore more of this in my subsequent posts.

My direction in 2008 for posts on this blog are:

1. Continue with my thoughts on Business Intelligence along Technology, Process and People dimensions
2. 
   
3. Provide a “Theory of Constraints” based view of BI with focus on strengthening the link between the 3 dimensions mentioned above.
4. 
   

Almost every interesting business area – Six Sigma, Balanced Scorecard, System Dynamics, Business Modeling, Enterprise Risk, Competitive Intelligence, etc. has its relationship with BI and we will see more of this in 2008.

Please do keep reading and share your thoughts as well Business Intelligence

BI Appliances

What is a BI Appliance?

If a data warehouse class database product or a reporting product or a data integration product or an all-in-one software package is pre installed and available in a preconfigured hardware box, then such a “hardware + software” box is called a ‘Business Intelligence'  Appliance’. The very purpose of an appliance model is to cover the underlying software components complexity and intricacies and make it simple like operating a TV system.

How an Appliance Model evolved?

As businesses gathered huge data, the demand for faster and better ways of analyzing data increased, the data warehouse as a software technology got evolved; there have been continuous efforts to build software systems that are cognizant of data warehouse environments.

We have seen IBM and Oracle releasing their data warehouse specific database editions

We seen the growth of data warehouse specific databases like RedBrick(now part of IBM), Teradata, Greenplum…
We have seen simple list reporting tools getting into proprietary data structures cubes and the emergence of acronyms MOLAP, HOLAP, ROLAP, DOLAP
We had a very new software market created for ETL and EII products
We have seen more new software applications related to BI being created BAM, CPM, Metadata Management, Data Discovery and lot more getting defined every day into the market….

As many organizations started setting up its BI infrastructure or enhanced its existing BI environment with different BI software packages they needed, they also imbibed different platforms and hardware, the maintenance of these became frightening. Getting started with a BI project by itself became a bigger project; we needed to spend sufficient time not just on choosing the right set of BI products but also on the supported hardware, dependent software packages and the platform. No BI vendor currently addresses the complete stack of BI system needs and this has been the driving factor for more acquisitions.

Products like Nettezza (Data base Appliance), CastIron (ETL Appliance) came up with their ‘software in a box’ concept, where we can buy or rent preconfigured ‘hardware + software’ boxes which in a way addresses the need of ‘ready to use’ BI market. Many of these boxes have Linux, open source databases, web server, message queues and proprietary software.

The Appliance based model is not new, IBM has been renting its ‘mainframe + software’ for decades. IBM has addressed the BI market with its ‘Balanced Warehouse’; a preconfigured ‘hardware + software’, its OS can vary from Windows – AIX – Linux with DB2 as database and data reporting can vary from DB2 Cubes – Crystal – Business Objects. HP in a similar way has come out with its Neoview platform which is a revitalized version of NonStop SQL database and NonStop OS.

The need of a CIO has been always ways to shorten the application deployment cycle and reduce the maintenance factor of the servers; the Appliance based products meet these KRA of a CIO and are getting accepted widely.

The Future

More Appliances, Focus on Performance:
We would see more BI appliances coming into market; as the Appliance model covers what’s underneath and in many cases the details being not available; the buying focus would be more on what the products deliver rather than what they have inside.

Common Appliance Standards:
Getting best of breed of software and hardware from a single vendor would not happen. We might see both software and hardware vendors defining a set of basic standards among themselves for the Appliance model. New organizations would also evolve similar to “tpc.org” which would define performance standards for appliances. We might see companies similar to DELL coming up which can assemble best of breed components and deliver a packaged BI Appliance.

More Acquisitions: The current  Business Intelligence Market landscape can also be interpreted as

1. Hardware + Software or Appliance based vendors – HP, IBM
2. Pure software or Non-Appliance based vendors – Oracle, Microsoft, SAP
3. 
   

Once the current BI software consolidation gets established the next wave of consolidation would be towards companies like Oracle looking for hardware companies to be added to their portfolio.

Technology Appliance Products Database Netezza Teradata DATAllegro Dataupia Data Integration CASTIron Reporting-Dashboard Cognos NOW (Celequest LAVA) Configurable Stack (with third party support) IBM Balanced Warehouse HP Neoview

“What Management Is” – The crucial link between Business and Intelligence

Let’s for a moment accept the hypothesis that the true intent of Business Intelligence is to help organizations manage their business better. “Better” in this context tends to be a rather elastic adjective as it straddles the entire spectrum of firms using BI for simple management reporting to the other extreme of using BI to ‘Compete on Analytics’ in the marketplace.

“Managing business better” presents the classic question of “What aspects of business can BI help manage better”. The Answer – “Pretty much everything”.

In this post, I would like to list down the different business areas that ought to be managed for the better and drill down into the applicability of BI for each of these areas in future posts. The primary reference for my listing is from one of the best management books I have ever read till date – “What Management Is” by Joan Magretta and Nan Stone. (http://www.amazon.com/What-Management-Works-Everyones-Business/dp/0743203186). This book really helps in drawing the boundaries around management concepts and for BI practitioners, like me, shows the direction for the evolution and business applicability of BI.

BI practitioners need to understand the following business areas:

1. Value Creation – BI can help in providing the critical “Outside-in” perspective
2. 
   
3. Business Model – Is this the right business to be in?
4. 
   
5. Strategy – Validation and tuning of Strategy thro’ BIsiness Intelligence
6. 
   
7. Organization Boundaries – BI can help solve the Build vs Buy conundrum
8. 
   
9. Numbers in Business – Really the sweetspot for BI applications
10. 
    
11. Mission and Measures – Connecting the company’s mission with the measures
12. 
    
13. Innovation and Uncertainty – Domain of Predictive Analytics & its ilk
14. 
    
15. Focus – Realm of Pareto’s Law vis-à-vis the more recent “Long-Tail” phenomenon
16. 
    
17. Managing People – Human Resource Analytics is one of the most happening analytics application areas at this point in time.

Bit of marketing here – after all, this is a corporate blog – My company Hexaware is a specialty provider of HR Analytics solutions. Please do visit –http://www.hexaware.com/new_hranalytic.htm for more information

To me, the list above presents the most comprehensive high-level thought process when confronted with implementation of BI in organizations. In my consulting engagements, the litmus test is to really see whether the BI strategy covers the different aspects of business as noted above 

– “More the coverage better is the BI vision”.

Information Nugget

I was quite fascinated by the range of analytical apps available “On-Demand” at http://www.salesforce.com/appexchange/category_list.jsp?NavCode__c=a0130000006P6IoAAK-1 . I personally feel that “On-Demand” does have the potential to disruptively change the way BI services have been delivered customers. More on that later!

The crucial link between Business and Intelligence

Have a Merry Christmas and a Happy New Year 2008!

Data Integration Challenge – Building Dynamic DI Systems – II

Following are the design aspects towards getting a DI system dynamic

1. Avoiding hard references, usage of parameter variables
2. 
   
3. Usage of lookup tables for code conversion
4. 
   
5. Setting and managing threshold value through tables
6. 
   
7. Segregating data processing logics into common reusable components
8. 
   
9. Ensuring that the required processes are controllable by the Business team with the required checks built in
10. 
    

We had defined the first two aspects in the earlier writing, let us look at the scenarios and approach for the other three items

Setting and managing threshold values through tables

In data validation process we also perform verification on the incoming data in terms of count or sum of a variable, in this case the validity of the count or sum derived is verified against a pre defined number usually called the ‘Threshold Value’. Some of the typical such validation are listed below

1. The number of new accounts created should not be more than 10% (Threshold Value) of the total records
2. 
   
3. The number of records received today and the number of records received yesterday can not vary by more than 250 records
4. 
   
5. The sum of the credit amount should not be greater than the 100000
6. 
   

This threshold value differs across data sources but in many cases the metric to be derived would be similar across the data sources. We can get these ‘threshold values’ into a relational table and integrate this ‘threshold’ table into the Data Integration Challenge process as a lookup table, this enables the same threshold based data validation code to implemented across different data sources and also apply the specific data source threshold value.

Segregating Data Processing Logics into Common Reusable Components

Having many reusable components in a system by itself makes a DI system dynamic or adaptable, the reason being that reusable components work on the basic aspect of parameterization of inputs and outputs of an existing process and parameterization is a key component to get a DI system dynamic. Some of the key characteristics to look for in a DI system that would help carve out a reusable component are

1. Multiple data sources providing data for a particular subject area like HR data coming from different HR systems
2. 
   
3. Same set of data being shared with multiple downstream systems or a data hub system
4. 
   
5. Existence of an industry standard format like SWIFT, HIPPA either as source or target
6. 
   
7. Integration with third party systems or their data like D&B, FairIsaac
8. 
   
9. Changing data layouts of the incoming data structure
10. 
    
11. Systems that capture survey data
12. 
    

Ensuring that the required processes are controllable by the Business team with the required checks built in

In many situations we are now seeing requirements where in the business would be providing regular inputs to the IT team of the DI systems, these are the situations where we can design and place the portions of the DI system parameters under the business control. Typical examples of such scenarios are

• In ‘threshold value’ based data validation, these values would be business driven i.e., ‘threshold table’ can be managed by the business team and they would be able to make changes to the threshold table without code changes and without IT support
• In many scenarios the invalid data would under go multiple passes and be need to be validated at different passes by the business in terms of starting a BI session, the input from the business could be just starting the process or as well providing input data
• The data to be pulled out from a warehouse based on a feed from an online application; a typical web service problem-solution

The need for the business team to control or feed the DI systems is common with companies that handle more external data as with market research firms and Software As A Service (SAAS) firms. The web service support from the leading Data Integration vendors plays a major role in full filing these needs.

BI Implementation Enabler: Calibration for BI systems

In the last 2 posts, we looked at the way Agile Framework can be implemented to manage BI systems. Diagram below is intended to reiterate the process of Agile Framework implementation with its Planning & Execution phases.

Having taking care of managing the evolution of enterprise BI systems, the next important aspect of implementation is “How to measure the evolution”? This brings us to the next important enabler – “Calibrating Business Intelligence   systems”

What is Calibration?

Calibration = “Measurement” – Can be defined as the alignment of process to certain calibration factors so that the health of the process can be measured with respect to those factors

How is Calibration used in the BI context?

• Strategic tool to prioritize and align the EDW with the corporate vision
• Measure the evolution of EDW against pre-set goals
• Mechanism to identify technology pain areas and take appropriate corrective actions
• Is a way to objectively communicate the progress of EDW to business stakeholders
• Helps the DW project manager in tactically planning for the immediate future

There are 3 levels of scorecards developed by Hexaware that helps in measuring the evolution of BI systems implemented using the Agile Framework.

The method of calibration and the usage of Analytic Hierarchy Process (AHP) are explained in the webinar available at http://www.hexaware.com/webcastarchive1.html . The title of the webinar is: Agile Framework for Calibrating the Enterprise Data Warehouse.

 You might want to read these awesome related posts Business Intelligence

BI Implementation Enabler: Agile Framework for Data Warehousing–Part 2

In my previous post, I established the fact that Agile Framework can be used to manage complexity in enterprise wide Business Intelligence systems.

Two phases to the Agile framework implementation are:

1. Planning Phase
2. 
   
3. Execution Phase
4. 
   

Agile Framework – Planning Phase

Planning is typically done at the end of a particular year for the subsequent year, once the business plans & budgets are finalized.

Assumptions / Pre-requisites

1. Enterprise BI infrastructure is already in place in the organization
2. 
   
3. Technology Architecture (BI Tools/Technologies) and Process Architecture (Standards, Policies, Procedures) are already defined.
4. 
   

Start –> End	Activities	Deliverables
Create & Prioritize the Stories	Conduct JAD sessions and collect user requirements Have stakeholders sign-off on BI vision document	Functionality List (Stories) – With approximate effort estimates
Create the phase plan	Identify Phases for completing the “Story”	“Story – Phase” Mapping Document
Identify the “Cycles”	Identify number of development & stabilization cycles required to complete the Phase	“Story – Phase – Cycles” Mapping Document
Create the Release Plan	Identify the cycles (across stories) that can fit into a particular release	Monthly Release Plan

Agile Framework – Execution Phase

Execution Phase is for implementing the monthly release. This has the following tasks:

Start –> End	Activities	Deliverables
Execute the Cycles	Each cycle will have its own specifications, design & test plan documents Develop the code to satisfy the requirements for each cycle	Design Document Test Plan Test Results
Deliver the Release	All the cycles combined into a working release (typically delivered once a month)	Code Release Plan
Deliver the Phase	When all cycles for particular phase are completed, perform a regression test on some of the critical cycles	Phase Release Plan
Complete the story	When all the phases for a particular story are completed, perform the regression test on some of the critical phases	Complete the documentation of the business functionality

I have seen lot of benefits in managing BI systems using the Agile framework, especially in Enterprise Data Warehousing situations.

Let me put some disclaimers here:

1. Agile Development with its brand of specific techniques around Xtreme Programming, Pair programming, SCRUM etc. is a vast area with its own body of knowledge & best practices. This post neither has the intention nor the author has the talent (!) to be an authoritative guide for Agile methodologies.
2. 
   
3. This post is aimed at Business Intelligence practitioners to stimulate a new way of thinking around managing complexity in Business Intelligence systems. This is definitely not a “silver bullet” for managing BI projects.
4. 
   

Information Nugget: Came across a wonderful blog titled “Occam’s Razor” by Mr. Avinash Kaushik (www.kaushik.net). Avinash is an authority on Web Analytics and his blogs have some great information for everybody. Happy reading!

BI Implementation Enabler: Agile Framework for Data Warehousing – Part 1

As part of the Business Intelligence Utopia series, I am going to focus on the implementation enablers in the next few posts. The first implementation enabler is: Agile Framework for Managing Business Intelligence systems

BI systems are complex to manage due to the following reasons:

• Keeps Evolving over time – Enterprise DW can never be completely built
• BI drives business decisions – Needs “Total Alignment” with corporate vision
• Power of BI applications increases exponentially as the number of information consumers increases
• Data Warehouses need to be measured & calibrated against pre-set goals
• Development & Support has to be managed concurrently

Standard process methodologies like the Waterfall model, Spiral & Iterative models are not suitable for managing Business Intelligence systems. One methodology I have seen work very well, having used it in multiple projects at Hexaware, is the “Agile Methodology”. The philosophy of the Agile framework fits in very nicely to alleviate some of the complex issues in managing BI systems.

Agile Methodology – Definition

Agile development is a software development approach that “cycles” through the development phases, from gathering requirements to delivering functionality into a working release.

Basic Tenets:

• Shared Vision and Small Teams working on specific functionality
• Frequent Releases that make business sense
• Relentlessly manage scope – Managing the scope-time-resources triad effectively
• Creating a Multi-Release Framework with master plan and supporting architecture
• Accommodate changes “gracefully”

BI practitioners would appreciate the fact that the basic tenets do provide solutions to some of the critical pain areas when it comes to managing enterprise BI systems.

The ultimate goal of any DW/BI project is to roll out new business functionality on a regular and rapid basis with a high degree of conformance to what is already there –> à Fits in well with the “Agile” philosophy

The next few posts will illustrate the practical application of the Agile Framework to Business Intelligence systems.

BI Information Nugget – One of the recent websites that I really liked is: www.biblogs.com. This is a Business Intelligence blog aggregation site that has blogs written by seasoned BI practitioners. Happy reading!

Data Integration Challenge – Building Dynamic DI Systems – I

One other challenge in a data integration project as it is with any other IT system is to build a data integration environment that is agile and flexible enough to accommodate the system changes related to business rules. The benefit of building a dynamic system is that the code changes are less frequently done or completely avoided in many cases, so it is very much a de-facto process to look for design opportunities to build a DI system that is dynamic.

A dynamic DI system accommodates variation in the incoming data and is able to respond without system failures or mal-data processing, in many such scenarios the DI environment also gets to be controlled by the Business Team with lesser support from the IT team.

Following are some of the design aspects towards getting a DI system dynamic

1. Avoiding hard references, usage of parameter variables
2. 
   
3. Usage of lookup tables for code conversion
4. 
   
5. Setting and managing threshold value through tables
6. 
   
7. Segregating data processing logics into common reusable components
8. 
   
9. Ensuring that the required processes are controllable by the Business team with the required checks built in
10. 
    

Avoiding hard references, usage of parameter variables:
In scenarios like defining simple database connections or to using a “Business Unit Number” for filtering data while extracting required data from a source system can be parameterized through variables. In the case of database parameterization it enables easier code movement across development-test-production environments and in the scenario of the Business Unit Number parameterization it enables running the same DI program for one other Business Unit with same data by just changing the parameter variable of Business unit Number

Usage of lookup tables for code conversion
In scenarios like where the incoming data value has to be converted to one other “standard value”, we usually write a IF..THEN… ELSE… syntax, here again we can bring in dynamism by having a lookup table which would carry the incoming value as one column and the standard value to be replaced as another column. The benefit is if there is any further change to the standard value the lookup table can be updated without opening the code and as well we can insert a new record into the lookup in case we need get the DI process handle a code conversion for a new incoming value.

We shall see more details and other design aspects in the coming days.

You might want to read these awesome related posts Data Integration Challenge

Business Intelligence Utopia – Implementation Enablers

In this series of posts on Business Intelligence Utopia, I have presented some thoughts around 5 key technology enablers that in my view will take BI to the next level. They are:

1. Proliferation of transactions systems that support the analytical feedback loop Real-time / Right-time

2.Data Integration

3.Data Governance

4.Service Oriented Architecture

5.Extensible Data Models

All of these put together answer the question of “Where do you want to see Business Intelligence in the future”? I have 5 more to go to complete the discussion on the “Power of Ten” key enablers.

At this point, I felt that the question of “How to make it happen?” is as important as the technology enablers and hence decided to take a bit of digression to focus on some implementation aspects for Business Intelligence Utopia. I call them the “Implementation Enablers”. These enablers focus on process methodologies, specific techniques etc. that helps in managing the evolution of Business Intelligence within your organization.

There are 3 implementation enablers of interest, at this point in time:

1. Agile Framework for BI
2. 
   
3. Business Intelligence Calibration
4. 
   
5. Function Point based Estimation technique for Data Warehousing
6. 
   

You can also visit the following link http://www.hexaware.com/webcastarchive1.html to listen to my recent webinar on “Agile Framework for Calibrating the Enterprise Data Warehouse” to get an idea of the implementation enablers.

To reiterate, my approach for future posts on this blog is to write about Business Intelligence Utopia along 2 dimensions:

Technology enablers – 10 of them have been identified. 5 of them were discussed already and 5 more to go.

Implementation enablers – 3 of them have been identified and will be elaborated in the next few weeks.

Bit of information – I recently took the TDWI (The Data Warehousing Institute) assessment of  Business Intelligence  Maturity done using their online tool and got some very good insights. If you are interested, please use the link- http://tdwi.org/display.aspx?id=8500 to take the assessment for your organization.

Data Integration Challenge – Understanding Lookup Process – III

In Part II we discussed ‘when to use’ and ‘when not to use’ the particular type of lookup process, the Direct Query lookup, Join based lookup and the Cache file based lookup. Now we shall see what are the points to be considered for better performance of these ‘lookup’ types.

In the case of Direct Query the following points are to be considered

• Index on the lookup condition columns
• Selecting only the required columns

In the case of Join based lookup, the following points are to be considered

• Index on the columns that are used as part of Join conditions
• Selecting only the required columns

In the case of Cache file based lookup, let us first try to understand the process of how these files are built and queried.

The key aspects of a Lookup Process are the

• SQL that pulls the data from lookup table
• Cache memory/files that holds the data
• Lookup Conditions that query the cache memory/file
• Output Columns that are returned back from the cache files

Cache file build process:

Based on the product Informatica or Datastage when a lookup process is being designed we would define the ‘lookup conditions’ or the ‘key fields’ and also define a list of fields that would need to be returned on lookup query. Based on these definitions the required data is pulled from lookup table and the cache file is populated with the data. The cache file structure is optimized for data retrieval assuming that the cache file would be queried based certain set of columns called ‘lookup conditions’ or ‘key fields’.

In the case of Informatica, the cache file is of separate index and data file, the index file has the fields that are part of the ‘lookup condition’ and the data file has the fields that are to be returned. Datastage cache files are called Hash files which are optimized based on the ‘key fields’.

Cache file query process:

Irrespective of the product of choice following would be the steps involved internally when a lookup process is invoked.

Process:

1. Get the Inputs for Lookup Query, Lookup Condition and Columns to be returned
2. Load the cache file to memory
3. 
   
4. Search the record(s) matching the Lookup condition values , in case of Informatica this search happens on the ‘index file’
5. 
   
6. Pull the required columns matching the condition and return, in case of Informatica with the result from ‘index file’ search, the data from the ‘data file’ is located and retrieved
7. 
   

In the search process, based on the memory availability there could be many disk hits and page swapping.

So in terms performance tuning we could look at two levels

1. how to optimize the cache file building process
2. 
   
3. how to optimize cache file query process
4. 
   

The following table lists the points to be considered for the better performance of a cache file based lookup

Category	Points to consider
Optimize Cache file building process	• While retrieving the records to build the cache file, sort the records by the lookup condition, this sorting would speed up the index (file) building process. This is because the search tree of the Index file would be built faster with lesser node realignment • Select only the required fields there by reducing the cache file size • Reusing the same cache file for multiple requirements for same or slightly varied lookup conditions
Optimize Cache file query process	• Sort the records that come from source to query the cache file by the lookup condition columns, this ensures less page swapping and page hits. If the subsequent input source records come in a continuous sorted order then the hits of the required index data in the memory is high and the disk swapping is reduced • Having a dedicated separate disk ensures a reserved space for the lookup cache files and also improves response of writing to the disk and reading from the disk • Avoid querying recurring lookup condition, by sorting the incoming records by the lookup condition

You might want to read these awesome related posts Data Integration Challenge