Building a sustainable DW2.0 program

In order to be successful with second generation data warehousing (DW2.0), organizations need to adopt a spiral development approach in which numerous small iterations of data warehouse development are accomplished frequently. In addition, it is advisable that organizations revisit their whole approach to managing their corporate data resources.

“The benefit of spiral development and the Seven Streams Approach™ is a regular cadence of delivery to the business based on quarterly releases, within the context of a sustainable architecture”
-Derek Strauss

Spiral Methodology – Overview of Key Features

The spiral methodology, illustrated below, is ideally suited for users who don't know what they want. Most of the time when a data warehouse project is undertaken, it is impossible to gather all the business requirements upfront. This is not due to anyone's fault – it's not because the business cannot make up its mind, and it's not because the information technology group is too out of touch to be able to understand the business' needs. It is simply the nature of the beast. Business intelligence capability is normally developed in a mode of discovery. The members of the business community will ultimately know what they want when they see it – and as soon as they get what they want, they realize that they want and need something additional.

Figure 1 – Spiral Development Methodology

Many people confuse the spiral methodology with the iterative approach that has its roots in the object-oriented disciplines. Although there are some similarities, the two methodologies are actually quite different.

Some of the hallmarks of the spiral development methodology are as follows:

• Spiral methodology makes extensive use of prototyping
• Major tasks in spiral methodology can occur in any sequence
• It is not necessary to wait for one task to finish before starting the next task
• A different project management philosophy and mindset is required
• Culture change in both the business and IT communities is required
• Expectations must be managed, because the first iteration of data warehouse development is incomplete and subject to further refinement
• A quarterly delivery cadence is typical, requiring strict scope discipline

The goal of spiral methodology is to build an inventory of reusable assets. Using the spiral methodology helps an organization to avoid endless "scrap and rework" cycles. It builds a culture based on a return on assets. This new way of thinking about things is designed to focus on reusable components that can be reassembled to meet the business' demand that things be done cheaper, faster and better.

Second generation data warehousing does not benefit from the conventional Return On Investment (ROI) approach. The latter approach has led many organizations to build successive "point solutions" (or data marts), with little or no attention given to the reusability of anything that has been delivered. The ROI imperative says: 'get it in as quickly as possible and get some return'. The fact that the initial success cannot be sustained, and the data structures prove to be brittle (not change-enabled) seems to escape the ROI mindset.

In the world of DW2.0, success is measured by ROA (return on assets). Has the data been reused? Has the metadata been reused? Have the structural business rules been reused? Is the ETL logic reusable and extensible? Are the components of the presentation layer reusable and flexible?

Introducing a spiral methodology is a critical step to success in second generation data warehousing. In addition, a whole new program approach to enterprise data needs to be adopted. This approach is outlined in the remainder of this editorial, encapsulated in a tried and tested approach called the Seven Streams Approach™.

Gavroshe's Seven Streams Approach™ – An Overview

Each of the seven streams in the DW/BI program approach focuses on a different aspect of the corporation's data architecture and is undertaken using  appropriate techniques and methods:

Stream 1 – Enterprise Reference Modeling is done subject-by-subject
Stream 2 – Enterprise Knowledge Co-coordination is done artifact-by-artifact
Stream 3 – Information Factory Development is done topic-by-topic
Stream 4 – Data Profiling and Mapping is done source-by-source
Stream 5 – Data Cleansing is done element-by-element
Stream 6 – Infrastructure Management is done component-by-component
Stream 7 – Total Quality data Management is done attribute-by-attribute

The key thing to note about the seven streams is the fact that each activity stream marches to the beat of a different drum. Each stream is simultaneously initiated, concurrently driven, and needs to be coordinated and monitored. There is no implied sequence in the way the diagram is organized.

Figure 2 – The Seven Streams Approach™ to DW/BI Programs

Each stream produces deliverables at different rates. The astute DW/BI Program Manager will recognize what these different rates and rhythms are, will synchronize the work priorities in each of these concurrent streams, and will use this information to define meaningful releases for the organization.

Summary

The Spiral Methodology has proven most effective for data warehouse projects. It is further enhanced by implementing the Seven Streams Approach™. Spiral DW/BI development iterations can be expedited if the correct level of attention is given to building the corporate data model, coordinating knowledge about the enterprise, doing proactive data profiling and mapping, doing proactive data cleansing, managing the infrastructure proactively, and establishing the correct culture for total quality data management. The combined impact of the method and the approach means that the development team will not hit as many roadblocks - data models, rules and definitions will be there waiting for them, data quality anomalies will be known way ahead of time and the appropriate infrastructure will be in place.

The benefit of spiral development and the Seven Streams Approach™ is a regular cadence of delivery to the business based on quarterly releases, within the context of a sustainable architecture. The challenge is the culture change required from both the IT and the business communities. The most successful organizations manage this culture change through a combination of training in spiral methodology and the Seven Streams Approach™, plus mentorship by people with deep skills and experience in both disciplines. The combined DW/BI program method and approach must also be supported by the correct governance structure.

Derek Strauss is founder, CEO and a Principal Consultant of Gavroshe. He has over 30 years of IT industry experience, including 24 years working in the Data Resource Management and Business Intelligence/ Data Warehousing fields. He is a co-author of the book "DW2.0 - The Architecture for the Next Generation of Data Warehousing", Morgan Kaufmann, 2008 (W.H. Inmon, D. Strauss, G. Neushloss).