Are You Managing Your BI Project Wrong? Waterfall vs. Agile Methodology
[Hint] If you’re using Waterfall project management, you’re doing it wrong. Deliver critical insights more quickly and with less effort and cost using Agile Scrum project management.
According to Gartner, Business Intelligence (BI) is an umbrella term for applications, tools, and best practices that enable access to and analysis of information that improves and optimizes outcomes.
As a business leader, there are lots of reasons why you may want your organization to embark on a BI project:
- You know you have the data; you just need tools in place to leverage it
- You have grown rapidly and need to consolidate data from multiple ERP systems and possibly more than one General Ledger
- Your reporting is manually intensive and time-consuming
- Reporting is causing performance problems or data security concerns with your critical transactional systems
- You are experiencing “Excel Hell” – your managers are spending more time gathering and formatting spreadsheets than they are using the information
- You desperately need a Single Source of Truth (SSOT)
- All of the above
All of these reasons share a common goal: To gain improved insights (actionable information) from data. In other words, you want to discover what you do not know. Fortunately, there are lots of data warehousing solutions, reporting tools, and analysis tools available to help you solve these challenges. You can also enlist any of the multitudes of technology and consulting firms out there (like Saxony Partners).
But on their own, none of these tools and partners can help you know what you do not know. This can only be discovered through insights. Insights that will naturally point you to the need to learn more.
Waterfall Project Management
Under waterfall project management, BI projects typically play out like this:
- Analysis Phase: Assess the current environment and identify tools and techniques to address problems
- Requirements Definition Phase: Document a list of reports and objectives
- Design Phase: Document what each report will look like, how users will interact with them
- Build Phase: Build the documented solution
- Test Phase: Ensure everything works as documented
Once all the routines, reports, and tools are tested – then development work is done. The tools are integrated into the production environment for the user base to begin using. Time to celebrate, right? Maybe not.
Notice that the determination of what needs to be built is decided at the start of a typical waterfall project. However, the reason for any BI project is to gain insight into what is not already known. As you gain that insight, you will want to know things you did not know you wanted to know. Unfortunately, you cannot know what you really need to build until after you have already built some pieces.
There is a better way to structure BI projects. Rather than waterfall project management, Agile principles and the Scrum software development framework embodies quick releases of production-ready deliverables.
Agile Project Management
You can read more about the Agile principles over on the Project Manager blog, but here is the shorthand:
- Customer satisfaction through early and continuous software delivery
- Accommodate changing requirements throughout the development process
- Deliver working software frequently
- Collaboration between the business stakeholders and developers throughout the project
- Support, trust, and motivate the people involved
- Enable face-to-face interactions
- The primary measure of progress is working software
- Agile processes to support a consistent development pace
- Attention to technical detail and design enhances agility
- Simplicity is key
- Self-organizing teams encourage great architectures, requirements, and designs
- Regular reflections on how to become more effective
These principles, combined with user-ready reports and tools, can help you quickly gain valuable insight. This initial insight will often point you to questions that you did not know to ask at the start of your BI project.
Speaking of reports and tools, we’ve noticed from past waterfall BI projects that many reports and tools have gone unused. Conversely, there were always a few key deliverables that were particularly insightful. The latter were surprises, more-or-less.
Under the Agile-based Scrum project framework, we would have discovered critical insights earlier. Doing so would have set the stage for future releases – and we would have saved the time and cost that went into building unused reports and tools. Our clients would have uncovered their desired insight quicker and more cheaply.
Another fundamental component of the Agile-based Scrum framework is delivering production-ready code within time-boxed intervals called sprints. Sprints should never be longer than two months; most project teams generally target a two-to-four-week cycle. Each sprint focuses on the highest-priority items that can be delivered in that sprint. Throughout each sprint, new items and priorities are identified – and these are carried over into subsequent sprints.
Such an approach is why the Agile Scrum framework works so well for BI projects. As you learn and gain insights from earlier sprints, those insights are used to drive what is built in subsequent sprints. The right tools and reports are consistently prioritized, again eliminating the time and cost otherwise spent on unused resources.