Patterns of Information Management

Use Best Practice Patterns to Understand and Architect Manageable, Efficient Information Supply Chains That Help You Leverage All Your Data and Knowledge

In the era of “Big Data,” information pervades every aspect of the organization. Therefore, architecting and managing it is a multi-disciplinary task. Now, two pioneering IBM® architects present proven architecture patterns that fully reflect this reality. Using their pattern language, you can accurately characterize the information issues associated with your own systems, and design solutions that succeed over both the short- and long-term.

Building on the analogy of a supply chain, Mandy Chessell and Harald C. Smith explain how information can be transformed, enriched, reconciled, redistributed, and utilized in even the most complex environments. Through a realistic, end-to-end case study, they help you blend overlapping information management, SOA, and BPM technologies that are often viewed as competitive.

Using this book’s patterns, you can integrate all levels of your architecture–from holistic, enterprise, system-level views down to low-level design elements. You can fully address key non-functional requirements such as the amount, quality, and pace of incoming data. Above all, you can create an IT landscape that is coherent, interconnected, efficient, effective, and manageable.

Coverage Includes

• Understanding how a pattern language can help you address key information management challenges
• Defining information strategy and governance for organizations and users
• Creating orderly information flows you can reuse and synchronize as needed
• Managing information structure, meaning, and lifecycles
• Providing for efficient information access and storage when deploying new IT capabilities
• Moving information efficiently and reliably to support your processes
• Determining how information should be processed and maintained
• Improving quality and accessibility, and supporting higher-value analytics
• Protecting information via validation, transformation, enrichment, correction, security, and monitoring
• Planning new information management projects in the context of your existing IT resources

• Understanding how a pattern language can help you address key information management challenges
• Defining information strategy and governance for organizations and users
• Creating orderly information flows you can reuse and synchronize as needed
• Managing information structure, meaning, and lifecycles
• Providing for efficient information access and storage when deploying new IT capabilities
• Moving information efficiently and reliably to support your processes
• Determining how information should be processed and maintained
• Improving quality and accessibility, and supporting higher-value analytics
• Protecting information via validation, transformation, enrichment, correction, security, and monitoring
• Planning new information management projects in the context of your existing IT resources

In the era of "Big Data," effective information management has become crucial to the success of virtually all organizations. Unfortunately, few IT practitioners know today's best practices for successfully managing enterprise information resources. Patterns for Information Management offers the solution: a multi-disciplinary patterns-based approach that reflects where information comes from, how it is distributed, protected, governed, monitored -- and, ultimately, utilized. Leading IBM information management experts Mandy Chessell and Harald Smith show how to design, integrate, and optimize "information supply chains" that:

* Effectively characterize the information issues associated with today's complex distributed systems

* Fully reflect today's service-oriented, message-based environments

* Establish an information landscape that is effective, manageable, ordered, and efficient.

* Create orderly flows of information which can be reused and synchronized wherever necessary

* Support reasoned decision-making about the applicability and consequences of each alternative approach

Chessell and Smith complement their plain-English information pattern language with a complete case study demonstrating its real-world applications. Their pioneering guidebook will serve as both an indispensable guide to architects at all levels of experience and an outstanding long-term information management reference.

Mandy Chessell

FREng CEng FBCS

Mandy has worked for IBM since 1987. She is an IBM Distinguished Engineer, IBM Master Inventor, and member of the IBM Academy of Technology Leadership Team. As the chief architect for InfoSphere® Solutions in IBM’s Software Group, Mandy designs common information integration patterns for different industries and solutions.

In earlier roles, Mandy’s work has focused on transaction processing, event management, business process management, information management, and model-driven development. This breadth is reflected in her invention portfolio, which to date stands at over 50 issued patents worldwide.

Outside of IBM, Mandy is a fellow of the Royal Academy of Engineering and a visiting professor at the University of Sheffield, UK. In 2001, she was the first woman to be awarded a Silver Medal by the Royal Academy of Engineering, and in 2000, she was one of the “TR100” young innovators identified by MIT’s Technology Review magazine. In 2006, she won a British Female Innovators and Inventors Network (BFIIN) “Building Capability” award for her work developing innovative people and the BlackBerry “2006 Best Woman in Technology - Corporate Sector” award. More recently, she was granted an honorary fellowship of the Institution for Engineering Designers (IED) and she won the “2012 everywoman Innovator of the Year.” For more information on Mandy’s publications, see http://en.wikipedia.org/wiki/Amanda_Chessell.

Harald Smith

Harald has worked for IBM since 2005. Harald is a software architect in IBM’s Software Group specializing in information quality, integration, and governance products, and is IBM certified in delivering IBM Information Management solutions. In this role, he develops best practices, methodology, and accelerators for common information integration use cases.

Harald has 30 years of experience working with data quality products and solutions; product and project management; application development and delivery; system auditing; technical services; and business processes across the software, financial services, healthcare, and education sectors. Harald was the product manager at Ascential® Software and IBM responsible for designing and bringing the IBM InfoSphere Information Analyzer product to market as a key component in IBM’s information quality portfolio. He has been issued three patents in the field of information quality and rule discovery and was recently recognized as an IBM developer- Works Contributing Author.

His publications include the IBM developerWorks articles “The information perspective of SOA design” [parts 6, 7, and 8], “Use IBM WebSphere® AuditStage in a federated database environment,” “Using pre-built rule definitions with IBM InfoSphere Information Analyzer,” “Designing an integration landscape with IBM InfoSphere Foundation Tools and Information Server” [part 1], and “Best practices for IBM InfoSphere Blueprint Director” [parts 1 and 2]. For the IBM InfoSphere Information Server documentation, Harald contributed to the “IBM InfoSphere Information Analyzer Methodology and Best Practices Guide” and “IBM Info- Sphere Information Server Integration Scenario Guide”; he has also contributed to three IBM Redbooks®.

About the Authors    xxix

Chapter 1  Introduction    1

Islands of Information    1

Introducing MCHS Trading    2

Improving an Organization’s Information Management    4

Patterns and Pattern Languages    7

Basic Components in the Pattern Language    11

Information Integration and Distribution    12

Pattern Language Structure    15

Summary    22

Chapter 2  The MCHS Trading Case Study    25

Introduction    25

Building an Information Strategy    26

Creating Management Reports    28

Creating a Single View of Product Details    30

Creating a Single View of Customer Details    37

Understanding the Status of Orders    44

Delivering Information Quality Improvements    47

Connecting MCHS Trading into a B2B Trading Partnership    51

Exploiting Predictive Analytics    55

Summary of Case Study    67

Chapter 3  People and Organizations    69

Information Centric Organization Patterns    70

Information Centric Organization    73

Information Management Obligation    78

Information Management Strategy    81

Information Management Principle    83

Information Governance Program    88

Information User Patterns    91

Information User    92

Variations of the Information User Pattern    95

Summary    97

Chapter 4  Information Architecture    99

Information Element Patterns    100

Information Element    101

Entity-Level Information Elements    106

Information Asset    107

Information Activity    110

Information Event    113

Information Processing Variables    114

Information Summary    117

Message-Level Information Elements    119

Information Payload    119

Attribute-Level Information Elements    122

Information Link    123

Information Metric    124

Information Code    126

Summary of Information Elements    129

Information Identification Patterns    129

Information Identification    130

Defining Which Information to Manage and How    133

Subject Area Definition    134

Valid Values Definition    136

Information Configuration    139

Defining How Information Is Structured    141

Information Model    141

Information Schema    143

Locating the Right Information to Use    145

Information Location    146

Semantic Tagging    148

Semantic Mapping    150

Different Reports About Information    153

Information Values Report    154

Information Values Profile    156

Information Lineage    159

Summary of Information Identification    161

Information Provisioning Patterns    161

Information Provisioning    163

Localized Provisioning    166

User Private Provisioning    167

Application Private Provisioning    169

Process-Level Provisioning    171

Daisy Chain Provisioning    172

Service-Level Provisioning    177

User Shared Provisioning    178

Service Oriented Provisioning    179

Linked Information Provisioning    182

Cache Provisioning    184

Collection-Level Provisioning    185

Snap Shot Provisioning    186

Mirroring Provisioning    189

Peer Provisioning    193

Event-Based Provisioning    196

Recovery Provisioning    197

Summary of Information Provisioning    199

Information Supply Chain Patterns    199

Information Supply Chain    200

Variations of the Information Supply Chain Pattern    209

Cascading Information Supply Chain    211

Hub Interchange Information Supply Chain    215

Single View Information Supply Chain    219

Consolidating Information Supply Chain    223

Hierarchical Information Supply Chain    225

Peer Exchange Information Supply Chain    230

Summary of Information Supply Chains    232

Summary    232

Chapter 5  Information at Rest    235

Information Service Patterns    236

Information Service    238

Information Service Implementation Patterns    243

Local Information Service    243

Remote Information Service    246

Triggering Information Service    250

Summary of Information Services    253

Information Collection Patterns    253

Information Collection    254

Location of Information    259

Physical Information Collection    260

Virtual Information Collection    263

Usage of an Information Collection    266

Master Usage    267

Reference Usage    271

Hybrid Usage    274

Sandbox Usage    277

Scope of an Information Collection    279

Complete Scope    280

Local Scope    282

Transient Scope    284

Coverage of an Information Collection    285

Complete Coverage    285

Core Coverage    286

Extended Coverage    286

Local Coverage    287

Summary of Information Collections    287

Information Entry Patterns    288

Information Entry    289

Identifying Information Using the Information Key    292

Information Key    295

Local Key    297

Recycled Key    300

Natural Key    302

Mirror Key    305

Aggregate Key    307

Caller’s Key    310

Stable Key    312

Structures for an Information Entry    316

Locking for an Information Entry    317

Specialized Operations for an Information Entry    317

Summary of Information Entries    318

Information Node Patterns    319

Information Node    320

Business Information Nodes    324

Application Node    326

Information Content Node    330

Search Node    332

Information Store    334

Information Mart    337

Information Cube    340

Integration Nodes    343

Information Broker    343

Queue Manager    349

Staging Area    353

Look-Up Table Node    356

Event Correlation Node    358

Operational Data Stores and Hubs    360

Information Mirror Store    361

Information Event Store    364

Operational Status Store    366

Information Asset Hub    368

Information Activity Hub    372

Big Data Information Processing Nodes    376

Information Warehouse    377

Streaming Analytics Node    380

Map-Reduce Node    382

Analysis Information Nodes    385

Information Analysis Node    386

Information Mining Store    389

Summary of Information Nodes    393

Summary    394

Chapter 6  Information in Motion    395

Information Request Patterns    396

Information Request    396

Variations of the Information Request Pattern    398

Summary of Information Requests    401

Information Flow Patterns    401

Information Flow    403

Routing Information Flows    406

Staged Routing    407

Partitioned Routing    410

Buffered Routing    413

Filtered Routing    416

Summarized Routing    418

Consolidating Information Flows    420

Synchronized Consolidation    421

Filtered Consolidation    424

Ordered Consolidation    427

Independent Consolidation    429

Distributing Information Flows    432

Synchronized Distribution    433

Partitioned Distribution    436

Ordered Distribution    439

Independent Distribution    441

Broadcast Distribution    444

Summary of Information Flow Patterns    446

Summary    447

Chapter 7  Information Processing    449

Information Trigger Patterns    450

Information Trigger    450

Variations of the Information Trigger Pattern    453

Manual Information Trigger    453

Scheduled Information Trigger    455

Information Service Trigger    458

Information Change Trigger    460

External Sensor Trigger    462

Summary of Information Triggers    465

Information Process Patterns    465

Information Process    466

Business Processes    469

Bespoke Application Process    471

Packaged Application Process    472

Agile Business Process    474

State Driven Process    476

Collaborative Editing Process    479

Information Reporting Process    481

Quality Information Processes    483

Information Profile Tracking Process    484

Clerical Review Process    487

Information Remediation Process    489

Information Validation Process    491

Information Matching Process    493

Information Ever-Greening Process    496

Information Archiving Process    497

Provisioning Information Processes    499

Information Replication Process    500

Information Deployment Process    502

Information Relocation Process    505

Information Federation Process    507

Information Queuing Process    509

Information Broadcasting Process    511

Information Summarizing Process    513

Information Scavenging Process    515

Analytics Processes    517

Information Pattern Discovery Process    518

Information Decision Definition Process    521

Information Pattern Detecting Process    523

Search Processes    525

IT Service Management Processes    526

Operational Health Monitoring Process    527

Information Node Management Process    528

Scheduling Process    530

Summary of Information Processes    531

Summary    532

Chapter 8  Information Protection    533

Information Reengineering Step Patterns    534

Information Reengineering Step    534

Specialist Information Reengineering Steps    538

Restructure Data    539

Standardize Data    542

Enrich Data    545

Classify Data    546

Check Data    548

Correct Data    550

Link Entries    553

Merge Entries    556

Separate Entries    558

Derive Value    560

Derive Relationship    562

Smooth Data    563

Sample Data    565

Information Guard Patterns    566

Information Guard    567

Variations of the Information Guard Pattern    570

Information Probe Patterns    572

Information Probe    572

Variations of the Information Probe Pattern    575

Summary    576

Chapter 9  Solutions for Information Management    577

Information Solution Patterns    578

Information Solution    578

Patterns for Changing Information Nodes    582

New Information Node    583

Information Node Upgrade    587

Patterns for Integrating Information Nodes    590

Distributed Activity Status    590

Semantic Integration    593

Partner Collaboration    597

Patterns for Master Data Management    599

Shared Master    601

Centralized Master    604

Information Registry    607

Golden Reference    610

Synchronized Masters    613

Patterns for Big Data and Warehouses    618

Historical System of Record    619

Workload Offload    624

Patterns for Business Intelligence and Analytics    626

Performance Reporting    626

Operational Analytics    628

Next Best Action    630

Patterns for Information Protection    633

Managed Archive    634

Information Access Audit    636

Information Monitoring    638

Summary of Information Solutions    642

Final Thoughts    642

Appendix 1  Glossary    643

Appendix 2  Summary of MCHS Trading’s Systems    649

Appendix 3  Related Pattern Languages    651

Appendix 4  Bibliography    653

Information Centric Organization    653

Enterprise Architecture    653

Enterprise Application Architecture    654

Service-Oriented Architecture and Web Services    655

Information Architecture    655

Data Models    655

Metadata Management    656

Information Warehouse    657

Information Integration    657

Information Quality    658

Master Data Management    659

Big Data    659

Analytics and Reporting    660

Information Security and Privacy    660

Information Life-Cycle Management    661

Patterns Index    663

Index    669