Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing

This is the eBook version of the printed book.

Using Agile methods, you can bring far greater innovation, value, and quality to any data warehouse, business intelligence, or analytics project. However, conventional Agile methodologies must be carefully adapted to address the unique characteristics of DW/BI projects. In "Agile Analytics," Agile pioneer Ken Collier shows how to do just that.

Collier introduces platform-agnostic Agile solutions for integrating infrastructures consisting of diverse operational, legacy, and specialty systems that mix commercial and custom code. Using working examples, he shows how to manage analytics development teams with widely diverse skill sets; support enormous and fast-growing data volumes; and more. Collieros techniques offer equal value whether your projects involve oback-endo data management, "o"front-endo business analysis, or both.

Part I focuses on Agile project management techniques and delivery team coordination, introducing core practices that shape the way your agile DW/BI project community works together towards success

Part II presents technical methods for enabling continuous delivery of business value at production-quality levels, including evolving superior designs; test-driven DW development; version control; and project automation

Collier brings together proven solutions you can apply right nownwhether youore an IT decision-maker, data warehouse professional, DBA, business intelligence specialist, or database developer. With his help, you can mitigate project risk, improve business alignment, achieve better resultsnand have fun along the way.

Ken Collier has worked with Agile methods since 2003, and pioneered the integration of Agile methods with data warehousing, business intelligence, and analytics to create the Agile Analytics style. He continues to refine these ideas as technical lead and project manager on several Agile DW/BI project teams. Collier frequently trains DW/BI teams in Agile Analytics, and has been a keynote speaker on the subject at HEDW (Higher Education Data Warehouse) 2011 and multiple TDWI (The Data Warehousing Institute) World Conferences. He is founder and president of KWC Technologies, Inc., and a senior consultant in the Cutter Consortium’s Agile Development and Business Intelligence practice areas.

Foreword by Jim Highsmith xv

Foreword by Wayne Eckerson xvii

Preface xix

Acknowledgments xxxiii

About the Author xxxv

Part I: Agile Analytics: Management Methods 1

Chapter 1: Introducing Agile Analytics 3

Alpine-Style Systems Development 4

What Is Agile Analytics? 7

Data Warehousing Architectures and Skill Sets 13

Why Do We Need Agile Analytics? 16

Introducing FlixBuster Analytics 22

Wrap-Up 23

Chapter 2: Agile Project Management 25

What Is Agile Project Management? 26

Phased-Sequential DW/BI Development 30

Envision → Explore Instead of Plan → Do 32

Changing the Role of Project Management 35

Making Sense of Agile “Flavors” 36

Tenets of Agility 39

Wrap-Up 56

Chapter 3: Community, Customers, and Collaboration 59

What Are Agile Community and Collaboration? 60

The Agile Community 64

A Continuum of Trust 67

The Mechanics of Collaboration 69

Consumer Collaboration 73

Doer Collaboration 77

Planner Collaboration 78

Precursors to Agility 80

Wrap-Up 82

Chapter 4: User Stories for BI Systems 85

What Are User Stories? 86

User Stories versus Requirements 89

From Roles to Use Cases to User Stories 92

Decomposing Epics 99

What’s the Smallest, Simplest Thing? 103

Story Prioritization and Backlog Management 107

Story-Point Estimating 111

Parking Lot Diagrams 117

Wrap-Up 119

Chapter 5: Self-Organizing Teams Boost Performance 121

What Is a Self-Organizing Team? 122

Self-Organization Requires Self-Discipline 127

Self-Organization Requires Shared Responsibility 128

Self-Organization Requires Team Working Agreements 130

Self-Organization Requires Honoring Commitments 132

Self-Organization Requires Glass-House Development 134

Self-Organizing Requires Corporate Alignment 136

Wrap-Up 137

Part II: Agile Analytics: Technical Methods 139

Chapter 6: Evolving Excellent Design 141

What Is Evolutionary Design? 144

How Much Up-Front Design? 148

Agile Modeling 149

Data Model Patterns 152

Managing Technical Debt 154

Refactoring 157

What Is Refactoring? 159

Deploying Warehouse Changes 167

Other Reasons to Take an Evolutionary Approach 171

Case Study: Adaptive Warehouse Architecture 174

Wrap-Up 189

Chapter 7: Test-Driven Data Warehouse Development 193

What Is Agile Analytics Testing? 194

Agile Testing Framework 197

BI Test Automation 201

Sandbox Development 211

Test-First BI Development 215

BI Testing Guidelines 220

Setup Time 221

Functional BI Testing 222

Wrap-Up 223

Chapter 8: Version Control for Data Warehousing 225

What Is Version Control? 226

The Repository 230

Working with Files 233

Organizing the Repository 240

Tagging and Branching 245

Choosing an Effective Tool 252

Wrap-Up 254

Chapter 9: Project Automation 257

What Is Project Automation? 258

Getting Started 261

Build Automation 262

Continuous Integration 274

Push-Button Releases 281

Wrap-Up 288

Chapter 10: Final Words 291

Focus on the Real Problem 291

Being Agile versus Doing Agile 293

Gnarly Problems 296

What about Emerging Technologies? 298

Adoption Strategies 299

Closing Thoughts . . . 306

References and Recommended Reading 309

Index 315