Microsoft BizTalk Server 2010 Unleashed (Unleashed)

Microsoft BizTalk Server 2010 Unleashed is the 100% new, 100% practical developer’s guide to Microsoft’s most powerful version of BizTalk Server. Written by an expert team of Microsoft insiders and BizTalk MVPs, it reflects unsurpassed experience with all phases of BizTalk enterprise solutions development, from planning through deployment and administration.

The authors begin by introducing BizTalk Server’s architecture and key integration concepts. Next, they offer in-depth coverage of BizTalk’s foundational features, including schemas, maps, orchestrations, and pipelines. You’ll learn how to effectively utilize both standard and custom adapters; monitor integration services; leverage cloud computing via Windows Azure; implement operational BI solutions; and make the most of business rules and BizTalk’s Business Rules Engine (BRE).

The authors offer best practices and “in the trenches” tips for everything from managing deployments through implementing state-of-the-art mobile RFID solutions. No other book offers this much useful, pragmatic, and tested knowledge for successful BizTalk development.

Brian Loesgen is a Principal Architect Evangelist on Microsoft’s Azure ISV team. A six-time Microsoft MVP, he has extensive experience in building advanced enterprise, ESB, and SOA solutions. He has coauthored eight books, including SOA with .NET and Windows Azure. Charles Young, a principal consultant at Solidsoft, and Jan Eliasen, an IT architect at Logica, have each been honored repeatedly as BizTalk MVPs, and are highly respected bloggers in the BizTalk community. Scott Colestock, chief architect for Trace Ventures, specializes in using BizTalk to implement integration and service orchestration solutions. He is a member of Microsoft’s Architectural Advisory Board and a BizTalk MVP. Anush Kumar, CTO of S3Edge, served as Microsoft’s RFID business leader and was heavily involved in designing and architecting its BizTalk RFID offerings. Jon Flanders is an independent consultant, instructor for Pluralsight, BizTalk MVP, and author of RESTful .NET.

• Apply BizTalk Server’s sophisticated, scalable message exchange model to support virtually any business requirement
• Get your schemas right the first time, so you can avoid major problems downstream
• Use BizTalk Mapper to create maps for transforming inbound and outbound XML messages and supporting business processes
• Use orchestration to automate even highly-complex processes
• Utilize adapters and WCF to integrate any application, service, or system
• Extend your application’s reach with Windows Azure AppFabric
• Capture key BizTalk infrastructure, services, and business metrics
• Build powerful rules-based solutions with Microsoft’s Business Rules Framework
• Establish an Enterprise Service Bus (ESB) that uses BizTalk Server as core messaging infrastructure
• Efficiently manage, configure, and troubleshoot BizTalk through the Administration Console
• Create RFID applications ranging from simple label printing to end-to-end business processes

The most complete, practical guide to BizTalk Server 2009: an all-new book focused on delivering real, start-to-finish enterprise solutions

Microsoft BizTalk Server 2009 Unleashed is the definitive, pragmatic guide to Microsoft's latest and most powerful version of BizTalk Server. In this book, a team of world-class BizTalk Server 2009 experts bring together the deep practical insights .NET developers need to solve real business problems with BizTalk Server 2009 in any enterprise environment. Drawing on their immense BizTalk experience, the authors present best practices for the entire development lifecycle, from planning and architecture through deployment, and beyond. Writing at just the right level of technical detail for experienced .NET developers now starting out with BizTalk, they cover these and many other crucial issues: " Architecting and designing effective, high-value BizTalk solutions " Working with BizTalk schemas, maps, orchestrations, pipelines, pipeline components, and adapters " Implementing business rules with the Microsoft Business Rules Framework " Creating highly-available, high-performance BizTalk environments " Monitoring business activity " Collaborating effectively among BizTalk developers and users " Using BizTalk's leading-edge RFID capabilities Note: This is a 100% new book, NOT an update to Microsoft BizTalk Server 2004 Unleashed.

Foreword    xxii

Part I  The Basics

1  What Is BizTalk Server?    3

A Brief History of Application Integration    3

BizTalk Server Capabilities    7

Adaptation    7

Mediation    8

Exception Handling    8

Orchestration and Choreography    9

Performance and Scalability    9

Security    10

Insight    10

Electronic Data Interchange    11

RFID Event Handling    11

What Is a “Typical” BizTalk Solution?    11

BizTalk Server, WCF, and WF    12

Summary    14

2  Schemas    15

BizTalk Schemas    16

XML Schema Definition    16

Properties    17

Internal Schemas    18

XML Schemas    20

Existing XSDs    20

Generating an XSD    21

Creating an XSD    21

Flat File Schemas    36

Add Existing Schemas    38

Creating by Hand    38

Flat File Schema Wizard    47

EDI Schemas    60

Messages That Are Not XML and Not Flat File    60

Pass-Through Pipeline    60

Custom Disassembler    61

Custom Editor Extensions    61

Third-Party Components    61

Property Promotion    61

Distinguished Fields    63

Promoted Properties    65

Property Demotion    66

When to Use What    67

Versioning of Schemas    69

No Long-Running Transactions and a Short Downtime Acceptable    69

Long-Running Transactions or a Short Downtime Is Unacceptable    70

Testing    71

Validate Schemas    71

Validate Instances    72

Generate Instances    74

Unit Testing of Schemas    75

Testing Using Pipeline Tools    80

Schemas for Scenario Used in This Book    81

FineFoods.Common.Schemas    81

FineFoods.CreditCheck.Schemas    82

FineFoods.Customers.C1701    82

FineFoods.Customers.C1702    83

FineFoods.Customers.Schemas    84

FineFoods.Inventory.Schemas    84

FineFoods.Orders.Schemas    84

FineFoods.PurchaseOrders.Schemas    87

Summary    88

3  Maps    89

The Mapper    90

Layout of Mapper    90

Initial Considerations    92

Creating a Simple Map    94

Functoids    108

String Functoids    111

Mathematical Functoids    112

Logical Functoids    113

Date/Time Functoids    115

Conversion Functoids    116

Scientific Functoids    116

Cumulative Functoids    117

Database Functoids    118

Advanced Functoids    120

Third-Party Functoids    122

Advanced Maps    123

Mapping Optional Fields    123

Looping Functoid    123

Index Functoid    125

Database Lookup    127

Scripting Functoid    129

Functoid Combination    131

Combination of Functoids for If-Then-Else    131

Create Separated List    132

Table Looping Functoid    132

Conditional Creation of Output Nodes    135

Custom XSLT    136

Cross Referencing    136

Building Custom Functoids    140

Initial Setup    141

Normal Functoid    146

Cumulative Functoid    151

Developing Advanced Functoids    155

Deployment of Custom Functoids    157

Debugging    161

Testing of Maps    163

Validating Maps    164

Testing Maps    164

Debugging a Map    167

Unit Testing    168

Summary    172

4  Orchestrations    173

Orchestration Designer    174

Defining Orchestrations    177

Building Orchestrations    178

Messages    182

Variables    186

Shapes    188

Delivery Notification and Handling Retries    217

Calling Pipelines    218

Web Services    221

Dehydration and Rehydration    228

Correlations    229

Convoys    234

Parallel Convoys    234

Sequential Convoys    235

Zombies    236

Transactions    237

Atomic    238

Long Running    240

Compensating Code    241

Persistence Points    246

Exception Handling    247

Debugging    250

Send Out Messages    250

Debug and Trace    250

Breakpoints in Orchestration Debugger    250

Summary    255

5  Pipelines    257

Stages in Pipelines    258

Stages in a Receive Pipeline    259

Stages in a Send Pipeline    261

Built-In Pipelines    262

Receive Pipelines    262

Send Pipelines    263

Built-In Pipeline Components    263

XML Components    264

Flat Files    268

Encoding, Encrypting, and Signing    272

BizTalk Framework    275

Validator and Party Resolution    280

Custom Pipelines    283

Using the Built-In Pipeline Templates    283

Creating Custom Pipeline Templates    284

Custom Pipeline Components    287

Resources, Attributes, and Constructors    288

Interfaces    292

Message and Context Interfaces    305

Miscellaneous Functionality    309

Streaming    314

Properties    317

Really Fast Pipeline Component Implementation    323

Deployment    324

Debugging    327

Pipeline Component Wizard    329

Testing    330

Pipeline    exe    330

Unit Testing    331

Summary    334

6  Adapters    337

BizTalk Adapters    337

Native Adapters    338

Line-of-Business Adapters    339

BizTalk Adapter Pack    339

Host Adapters    339

Third-Party and Custom Adapters    339

Additional Microsoft Adapters    340

The Role of WCF Adapters    340

Adapter Characteristics    340

Direction    341

Push and Pull    341

Message Interchange Pattern    341

Hosting    342

Configuration    342

Batches    343

Transactions    344

Message Context    344

Metadata Harvesting    344

Registering Adapters    345

Creating Adapter Handlers    346

Port-Level Configuration    349

Configuring Receive Locations    350

Configuring Send Ports    352

Adapter Properties    355

Deploying Bindings    355

Native Adapters    357

File Adapter    357

Robust Interchange    357

Polling Locked Files    358

File Renaming    359

Reliable Messaging Issues    359

Path and File Names    359

Security    360

Additional Send Handler Issues    360

FTP Adapter    360

FTP Issues    361

Handling Duplicate Messages    362

Staging Files in Temporary Folders    362

Raw FTP Commands    363

Secure Messaging    363

HTTP Adapter    364

Using HTTP Receive Handlers    364

Using HTTP Send Handlers    366

Additional Configuration    366

MQ Series Adapter    367

Using MQ Series Receive Handlers    368

Using MQ Series Send Handlers    369

Managing Queues    369

Configuring MQSAgent    370

MSMQ Adapter    370

Using MSMQ Receive Handlers    371

Using MSMQ Send Handlers    372

Authenticating and Securing Messages    374

POP3 Adapter    375

Using POP3 Receive Handlers    376

Handling Encrypted Messages    377

SMTP Adapter    377

Using SMTP Send Handlers    378

Windows SharePoint Services Adapter    379

Using WSS Receive Handlers    380

Using WSS Send Handlers    381

Mapping SharePoint Columns    383

SOAP Adapter    383

WCF Adapters    384

Windows Communication Foundation    385

Comparing WCF to BizTalk Server    386

The Role of BizTalk Native WCF Adapters    388

Hosting Native WCF Adapters    389

The WCF Service Publishing Wizard    389

Publishing Orchestrations    392

Publishing Schemas    392

WCF Send Handlers    394

Importing MEX Endpoints    395

Importing Metadata Files    396

Dynamic Ports    397

Configuring WCF Adapters    397

Addresses and Identity    398

Bindings    399

Behavior    400

Security and Credentials    401

Message Handling    402

Using the SQL Server LoB Adapter    404

WCF LoB Framework and SDK    404

SQL Server Adapter    404

Polling and Notification    405

Performing Operations via a Send Handler    407

Additional Adapter Capabilities    408

Metadata Harvesting    409

Summary    412

Part II  Advanced Topics

7  BizTalk 2010 and WCF: Extensibility    415

WCF Extensibility    416

The WCF Channel Stack    416

ABCs Reviewed    417

ServiceContract in BizTalk    418

WCF Behaviors    420

Example of WCF Extensibility in BizTalk    420

Summary    429

8  BizTalk and Windows Azure    431

Extending the Reach of BizTalk Applications    431

The AppFabric SDK    432

Receiving Messages    433

Sending Messages    434

Static Send Port    435

Dynamic Send Port    436

ESB Off-Ramp    436

Using InfoPath as a Client    438

Summary    439

9  Business Activity Monitoring with BizTalk BAM    441

BAM and Metrics    441

What Is BizTalk BAM?    442

Using BizTalk BAM    444

End-to-End, High-Level Walkthrough of the BAM Process    444

Real-Time Versus Scheduled Aggregations    446

Defining Activities and Views    447

Progress Dimension    450

Data Dimension    450

Numeric Range Dimension    450

Time Dimension    450

Using the Tracking Profile Editor    452

Using the BAM APIs    453

DirectEventStream (DES)    453

BufferedEventStream (BES)    453

OrchestrationEventStream (OES)    454

IPipelineContext Interface    454

Creating a Higher-Level API Specifically for Service Metrics    454

Working with the WCF and WF Interceptors    457

Using Notifications    460

Rapid Prototyping    460

REST and BAM    461

Managing BAM    461

BAM Database Considerations    461

Deployment and Management    461

Security    462

Scripting Deployment    462

Summary    465

10  The Business Rules Framework    467

The Importance of Rules    468

Processes and Policies    468

Business Policies    469

Policy Externalization    469

Policy Scenarios    471

Business Versus Executable Rules    472

Business Rule Management    473

BRMS and the BRF    475

Example Scenario: Order Processing    476

Incomplete and Implicit Business Rules    478

Indirect Policy Mapping    478

Technical Policy    479

Data Models    479

Programmatic Bindings    479

Priority    479

Traceability    479

Refactoring    480

Testing, Publishing, and Deployment    480

Managing Change    481

Real-World Rule-Processing    482

Using Vocabularies    483

What About Performance?    484

Inference and Reasoning    485

The Business Rules Framework    487

Introducing the BRF    487

Rule Storage and Administration    488

Rule Deployment    489

Rule Modeling    495

Rule Execution    497

Components and Tools    499

Microsoft Business Rule Language    499

Business Rules Database    499

Pub-Sub Adapter    504

Rule Store Components    505

SqlRuleStore    505

OleDbRuleStore    505

FileRuleStore    506

Rule Set Deployment Driver    507

Business Rules Language Converter    507

Business Rules Engine    507

Policy Class    507

Policy Tester Class    509

BizTalk Server 2010 Rule Engine Extensions    511

Rule Definition and Deployment    511

The Rule Composer    512

Loading Rule Stores    513

Using the Policy Explorer    516

Using the Facts Explorer    520

Composing Rule Conditions    525

Creating Rule Actions    530

Rule Engine Component Configuration    532

Testing Rule Sets    534

Vocabularies    538

Strategies for Vocabulary Versioning    543

Publishing and Deployment    545

The Rules Engine Deployment Wizard    546

Using Rules with BizTalk Server    547

ESB Toolkit    547

RFID Server    548

Using Rules in Custom Code    548

Policy Management in the Administration Console    548

The Call Rules Orchestration Shape    551

Policy-Driven Features of the ESB Toolkit    556

The RFID Server BRE Event Handler    558

Summary    561

11  Rule-Based Programming    563

The Roots of Confusion    563

Declarativity    564

Set-Based Programming    565

Recursive Processing    565

Blended Paradigms    566

Limits of Expressivity    567

Understanding the Rule Engine    568

Understanding Production Systems    568

Understanding Short-Circuiting    571

Using OR Connectives    573

Understanding Implicit Conditions    576

Common Rule Patterns    577

Implementing Quantification    577

Handling Negation-as-Failure    581

Using Strong Negation    583

Designing Rule Sets as State Machines    584

Exploiting Situated Reasoning    587

Rule Engine Mechanisms    589

Understanding Working Memory    589

The Match-Resolve-Act Cycle    590

Introducing the Rete Algorithm    593

Managing Conflict Resolution    594

Forward- and Backward-Chaining    595

Working with Facts    597

Using Typed Fact Classes    597

Handling XML Documents    598

Setting XPath Properties in the Rule Composer    599

XML Type Specifiers    600

Handling XML Namespaces    602

Reading and Writing XML Data    602

Managing Optional XML Nodes    603

Handling ADO    NET DataTable and DataRow Objects    606

Handling Data Connections    607

Handling    NET Types    609

Invoking Static Type Members    613

Optimizing Rule Sets    615

Controlling Side Effects    615

Optimizing the Rete Network    617

Programming with the Rule API    618

Using the Policy Class    618

Handling Long-Term Facts    623

Implementing Compensation Handlers    624

Using the RuleEngine Class    627

Implementing Custom Rule Store Components    628

Managing Deployment Programmatically    630

Creating Rules Programmatically    633

Summary    637

12  ESB with BizTalk Server    639

What Is an ESB?    639

Introducing the Enterprise Service Bus    639

What Problems Does an ESB Solve?    640

What Are the Components of an ESB?    641

Dynamic Routing    643

Dynamic Transformation    644

Message Validation    644

Message-Oriented Middleware    645

Is BizTalk a Fully Functional ESB?    645

What Is the ESB Toolkit?    645

History of the ESB Toolkit    646

What Is in the ESB Toolkit?    646

What’s the Difference Between Native BizTalk Server and BizTalk Server with the ESB Toolkit?    646

The Magic Behind an ESB    647

The ESB Toolkit Stack    649

Itineraries    650

Specifying Itineraries    651

The Itinerary Lifecycle    652

Dynamic Resolution: The Resolvers    653

Adapter Providers    655

Service Composition    656

Messaging-Only Implementations    657

Unified Exception Management    658

Exposing Core Services    660

Distributed ESBs    660

REST and BizTalk ESB    661

A Stylistic Comparison    661

Incorporating REST into the BizTalk ESB    662

Management    662

Provisioning and Runtime Governance    662

SLA Enforcement    663

Monitoring    663

Organizational Considerations    664

Ensuring a Smooth Transition    664

Gatekeeper Process    665

Summary    666

Part III  Deployment and Administration

13  Administration Console Concepts    669

Introducing the Administration Console    669

BizTalk Group Properties    670

BizTalk Settings Dashboard    672

Group Hub and Query View    678

Applications Node    680

Platform Settings    681

Hosts    681

Host Instances    681

Servers    682

Message Boxes    683

Adapters    684

Summary    685

14  Deployment Concepts    687

The Work to Be Done    687

“Application” as a Formal Concept    689

Where Does It All Begin? (Inside Visual Studio)    691

Folder and Project Structure    691

Namespaces and Assembly Names    692

Applying Strong Names    693

Setting Deployment Properties    694

Fine Foods Solution    696

Deploying from Visual Studio    697

Binding and Starting the Application    698

Edit/Debug Cycle    700

Handling Binding Files During Development    703

Creating and Managing Deployable Packages    704

Other Types of Resources    707

Binding Files as Resources    708

Deployment Scripts as Resources    709

Exporting MSI Files    712

Handling MSI Export on a Build Server    713

Deploying MSI Packages to a BizTalk Group    715

Import/Install via Command Line    717

Handling Other Deployables    718

Business Activity Monitoring    718

Rule Vocabularies and Policies    719

Handling Upgrade and Versioning Scenarios    719

Summary    720

Part IV  RFID

15  BizTalk RFID    723

RFID Overview    724

The BizTalk RFID Framework    725

Installation Notes for BizTalk RFID    727

Device Applications    731

Vendor Extensions and Extensibility    743

Tag Operations    749

Introducing RFID Processes    756

Exception Handling    771

Debugging (Process Hosting Model)    773

Integration and Deployment Considerations    773

Summary    778

16  BizTalk RFID Mobile    779

Mobile RFID Overview    779

The BizTalk RFID Mobile Framework    780

Installation Notes    781

Device Applications    782

Running Your First Mobile Application    787

Barcode Support    791

BizTalk RFID Mobile Connector Architecture (Store and Forward)    792

Remote Device Management    796

Summary    798

Closing notes    799

Index    803