BI Implementation Enabler: Calibration for BI systems

In the last 2 posts, we looked at the way Agile Framework can be implemented to manage BI systems. Diagram below is intended to reiterate the process of Agile Framework implementation with its Planning & Execution phases.

Having taking care of managing the evolution of enterprise BI systems, the next important aspect of implementation is “How to measure the evolution”? This brings us to the next important enabler – “Calibrating Business Intelligence   systems”

What is Calibration?

Calibration = “Measurement” – Can be defined as the alignment of process to certain calibration factors so that the health of the process can be measured with respect to those factors

How is Calibration used in the BI context?

• Strategic tool to prioritize and align the EDW with the corporate vision
• Measure the evolution of EDW against pre-set goals
• Mechanism to identify technology pain areas and take appropriate corrective actions
• Is a way to objectively communicate the progress of EDW to business stakeholders
• Helps the DW project manager in tactically planning for the immediate future

There are 3 levels of scorecards developed by Hexaware that helps in measuring the evolution of BI systems implemented using the Agile Framework.

The method of calibration and the usage of Analytic Hierarchy Process (AHP) are explained in the webinar available at http://www.hexaware.com/webcastarchive1.html . The title of the webinar is: Agile Framework for Calibrating the Enterprise Data Warehouse.

 You might want to read these awesome related posts Business Intelligence

BI Implementation Enabler: Agile Framework for Data Warehousing–Part 2

In my previous post, I established the fact that Agile Framework can be used to manage complexity in enterprise wide Business Intelligence systems.

Two phases to the Agile framework implementation are:

1. Planning Phase
2. 
   
3. Execution Phase
4. 
   

Agile Framework – Planning Phase

Planning is typically done at the end of a particular year for the subsequent year, once the business plans & budgets are finalized.

Assumptions / Pre-requisites

1. Enterprise BI infrastructure is already in place in the organization
2. 
   
3. Technology Architecture (BI Tools/Technologies) and Process Architecture (Standards, Policies, Procedures) are already defined.
4. 
   

Start –> End	Activities	Deliverables
Create & Prioritize the Stories	Conduct JAD sessions and collect user requirements Have stakeholders sign-off on BI vision document	Functionality List (Stories) – With approximate effort estimates
Create the phase plan	Identify Phases for completing the “Story”	“Story – Phase” Mapping Document
Identify the “Cycles”	Identify number of development & stabilization cycles required to complete the Phase	“Story – Phase – Cycles” Mapping Document
Create the Release Plan	Identify the cycles (across stories) that can fit into a particular release	Monthly Release Plan

Agile Framework – Execution Phase

Execution Phase is for implementing the monthly release. This has the following tasks:

Start –> End	Activities	Deliverables
Execute the Cycles	Each cycle will have its own specifications, design & test plan documents Develop the code to satisfy the requirements for each cycle	Design Document Test Plan Test Results
Deliver the Release	All the cycles combined into a working release (typically delivered once a month)	Code Release Plan
Deliver the Phase	When all cycles for particular phase are completed, perform a regression test on some of the critical cycles	Phase Release Plan
Complete the story	When all the phases for a particular story are completed, perform the regression test on some of the critical phases	Complete the documentation of the business functionality

I have seen lot of benefits in managing BI systems using the Agile framework, especially in Enterprise Data Warehousing situations.

Let me put some disclaimers here:

1. Agile Development with its brand of specific techniques around Xtreme Programming, Pair programming, SCRUM etc. is a vast area with its own body of knowledge & best practices. This post neither has the intention nor the author has the talent (!) to be an authoritative guide for Agile methodologies.
2. 
   
3. This post is aimed at Business Intelligence practitioners to stimulate a new way of thinking around managing complexity in Business Intelligence systems. This is definitely not a “silver bullet” for managing BI projects.
4. 
   

Information Nugget: Came across a wonderful blog titled “Occam’s Razor” by Mr. Avinash Kaushik (www.kaushik.net). Avinash is an authority on Web Analytics and his blogs have some great information for everybody. Happy reading!

BI Implementation Enabler: Agile Framework for Data Warehousing – Part 1

As part of the Business Intelligence Utopia series, I am going to focus on the implementation enablers in the next few posts. The first implementation enabler is: Agile Framework for Managing Business Intelligence systems

BI systems are complex to manage due to the following reasons:

• Keeps Evolving over time – Enterprise DW can never be completely built
• BI drives business decisions – Needs “Total Alignment” with corporate vision
• Power of BI applications increases exponentially as the number of information consumers increases
• Data Warehouses need to be measured & calibrated against pre-set goals
• Development & Support has to be managed concurrently

Standard process methodologies like the Waterfall model, Spiral & Iterative models are not suitable for managing Business Intelligence systems. One methodology I have seen work very well, having used it in multiple projects at Hexaware, is the “Agile Methodology”. The philosophy of the Agile framework fits in very nicely to alleviate some of the complex issues in managing BI systems.

Agile Methodology – Definition

Agile development is a software development approach that “cycles” through the development phases, from gathering requirements to delivering functionality into a working release.

Basic Tenets:

• Shared Vision and Small Teams working on specific functionality
• Frequent Releases that make business sense
• Relentlessly manage scope – Managing the scope-time-resources triad effectively
• Creating a Multi-Release Framework with master plan and supporting architecture
• Accommodate changes “gracefully”

BI practitioners would appreciate the fact that the basic tenets do provide solutions to some of the critical pain areas when it comes to managing enterprise BI systems.

The ultimate goal of any DW/BI project is to roll out new business functionality on a regular and rapid basis with a high degree of conformance to what is already there –> à Fits in well with the “Agile” philosophy

The next few posts will illustrate the practical application of the Agile Framework to Business Intelligence systems.

BI Information Nugget – One of the recent websites that I really liked is: www.biblogs.com. This is a Business Intelligence blog aggregation site that has blogs written by seasoned BI practitioners. Happy reading!

Data Integration Challenge – Building Dynamic DI Systems – I

One other challenge in a data integration project as it is with any other IT system is to build a data integration environment that is agile and flexible enough to accommodate the system changes related to business rules. The benefit of building a dynamic system is that the code changes are less frequently done or completely avoided in many cases, so it is very much a de-facto process to look for design opportunities to build a DI system that is dynamic.

A dynamic DI system accommodates variation in the incoming data and is able to respond without system failures or mal-data processing, in many such scenarios the DI environment also gets to be controlled by the Business Team with lesser support from the IT team.

Following are some of the design aspects towards getting a DI system dynamic

1. Avoiding hard references, usage of parameter variables
2. 
   
3. Usage of lookup tables for code conversion
4. 
   
5. Setting and managing threshold value through tables
6. 
   
7. Segregating data processing logics into common reusable components
8. 
   
9. Ensuring that the required processes are controllable by the Business team with the required checks built in
10. 
    

Avoiding hard references, usage of parameter variables:
In scenarios like defining simple database connections or to using a “Business Unit Number” for filtering data while extracting required data from a source system can be parameterized through variables. In the case of database parameterization it enables easier code movement across development-test-production environments and in the scenario of the Business Unit Number parameterization it enables running the same DI program for one other Business Unit with same data by just changing the parameter variable of Business unit Number

Usage of lookup tables for code conversion
In scenarios like where the incoming data value has to be converted to one other “standard value”, we usually write a IF..THEN… ELSE… syntax, here again we can bring in dynamism by having a lookup table which would carry the incoming value as one column and the standard value to be replaced as another column. The benefit is if there is any further change to the standard value the lookup table can be updated without opening the code and as well we can insert a new record into the lookup in case we need get the DI process handle a code conversion for a new incoming value.

We shall see more details and other design aspects in the coming days.

You might want to read these awesome related posts Data Integration Challenge

Business Intelligence Utopia – Implementation Enablers

In this series of posts on Business Intelligence Utopia, I have presented some thoughts around 5 key technology enablers that in my view will take BI to the next level. They are:

1. Proliferation of transactions systems that support the analytical feedback loop Real-time / Right-time

2.Data Integration

3.Data Governance

4.Service Oriented Architecture

5.Extensible Data Models

All of these put together answer the question of “Where do you want to see Business Intelligence in the future”? I have 5 more to go to complete the discussion on the “Power of Ten” key enablers.

At this point, I felt that the question of “How to make it happen?” is as important as the technology enablers and hence decided to take a bit of digression to focus on some implementation aspects for Business Intelligence Utopia. I call them the “Implementation Enablers”. These enablers focus on process methodologies, specific techniques etc. that helps in managing the evolution of Business Intelligence within your organization.

There are 3 implementation enablers of interest, at this point in time:

1. Agile Framework for BI
2. 
   
3. Business Intelligence Calibration
4. 
   
5. Function Point based Estimation technique for Data Warehousing
6. 
   

You can also visit the following link http://www.hexaware.com/webcastarchive1.html to listen to my recent webinar on “Agile Framework for Calibrating the Enterprise Data Warehouse” to get an idea of the implementation enablers.

To reiterate, my approach for future posts on this blog is to write about Business Intelligence Utopia along 2 dimensions:

Technology enablers – 10 of them have been identified. 5 of them were discussed already and 5 more to go.

Implementation enablers – 3 of them have been identified and will be elaborated in the next few weeks.

Bit of information – I recently took the TDWI (The Data Warehousing Institute) assessment of  Business Intelligence  Maturity done using their online tool and got some very good insights. If you are interested, please use the link- http://tdwi.org/display.aspx?id=8500 to take the assessment for your organization.