Synopses & Reviews
Despite its growing use in the enterprise, building applications for Hadoop is notoriously difficult. But there is a solution. This hands-on book introduces you to Cascading, the framework that enables you to build powerful data processing applications on Hadoop without having to spend months learning the intricacies of MapReduce.
Whether youre a developer, data scientist, or system/IT administrator, youll quickly learn Cascadings streamlined approach to data processing, data filtering, and workflow optimization, using sample apps based on Java, Scala, and Clojure. Companies such as Etsy, Razorfish, TeleNav, and Twitter already use Cascading for mission-critical applications. This book shows you how this framework can help your organization extract meaningful information from large amounts of distributed data.
- Examine best practices for using data science in enterprise-scale apps
- Learn how to use workflows that reach beyond MapReduce to integrate other popular Big Data frameworks
- Quickly build and test applications with familiar constructs and reusable components, and instantly deploy them onto large clusters
- Easily discover, model, and analyze both unstructured and semi-structured data in any format and from any source
- Seamlessly move and scale application deployments from development to production, regardless of cluster location or data size
Synopsis
There is an easier way to build Hadoop applications. With this hands-on book, youll learn how to use Cascading, the open source abstraction framework for Hadoop that lets you easily create and manage powerful enterprise-grade data processing applications—without having to learn the intricacies of MapReduce.
Working with sample apps based on Java and other JVM languages, youll quickly learn Cascadings streamlined approach to data processing, data filtering, and workflow optimization. This book demonstrates how this framework can help your business extract meaningful information from large amounts of distributed data.
- Start working on Cascading example projects right away
- Model and analyze unstructured data in any format, from any source
- Build and test applications with familiar constructs and reusable components
- Work with the Scalding and Cascalog Domain-Specific Languages
- Easily deploy applications to Hadoop, regardless of cluster location or data size
- Build workflows that integrate several big data frameworks and processes
- Explore common use cases for Cascading, including features and tools that support them
- Examine a case study that uses a dataset from the Open Data Initiative
About the Author
Paco Nathan is a Data Scientist at Concurrent, Inc., and heads up the developer outreach program there. He has a dual background from Stanford in math/stats and distributed computing, with 25+ years experience in the tech industry. As an expert in Hadoop, R, predictive analytics, machine learning, natural language processing, Paco has built and led several expert Data Science teams, with data infrastructure based on large-scale cloud deployments. He has presented twice on the AWS Start-Up Tour, and gives talks often about Hadoop, Data Science, and Cloud Computing.
Table of Contents
Preface; Requirements; Enterprise Data Workflows; Complexity, More So Than Bigness; Origins of the Cascading API; Using Code Examples; Safari® Books Online; How to Contact Us; Kudos; Chapter 1: Getting Started; 1.1 Programming Environment Setup; 1.2 Example 1: Simplest Possible App in Cascading; 1.3 Build and Run; 1.4 Cascading Taxonomy; 1.5 Example 2: The Ubiquitous Word Count; 1.6 Flow Diagrams; 1.7 Predictability at Scale; Chapter 2: Extending Pipe Assemblies; 2.1 Example 3: Customized Operations; 2.2 Scrubbing Tokens; 2.3 Example 4: Replicated Joins; 2.4 Stop Words and Replicated Joins; 2.5 Comparing with Apache Pig; 2.6 Comparing with Apache Hive; Chapter 3: Test-Driven Development; 3.1 Example 5: TF-IDF Implementation; 3.2 Example 6: TF-IDF with Testing; 3.3 A Word or Two About Testing; Chapter 4: Scalding--A Scala DSL for Cascading; 4.1 Why Use Scalding?; 4.2 Getting Started with Scalding; 4.3 Example 3 in Scalding: Word Count with Customized Operations; 4.4 A Word or Two about Functional Programming; 4.5 Example 4 in Scalding: Replicated Joins; 4.6 Build Scalding Apps with Gradle; 4.7 Running on Amazon AWS; Chapter 5: Cascalog--A Clojure DSL for Cascading; 5.1 Why Use Cascalog?; 5.2 Getting Started with Cascalog; 5.3 Example 1 in Cascalog: Simplest Possible App; 5.4 Example 4 in Cascalog: Replicated Joins; 5.5 Example 6 in Cascalog: TF-IDF with Testing; 5.6 Cascalog Technology and Uses; Chapter 6: Beyond MapReduce; 6.1 Applications and Organizations; 6.2 Lingual, a DSL for ANSI SQL; 6.3 Pattern, a DSL for Predictive Model Markup Language; Chapter 7: The Workflow Abstraction; 7.1 Key Insights; 7.2 Pattern Language; 7.3 Literate Programming; 7.4 Separation of Concerns; 7.5 Functional Relational Programming; 7.6 Enterprise vs. Start-Ups; Chapter 8: Case Study: City of Palo Alto Open Data; 8.1 Why Open Data?; 8.2 City of Palo Alto; 8.3 Moving from Raw Sources to Data Products; 8.4 Calibrating Metrics for the Recommender; 8.5 Spatial Indexing; 8.6 Personalization; 8.7 Recommendations; 8.8 Build and Run; 8.9 Key Points of the Recommender Workflow; Troubleshooting Workflows; Build and Runtime Problems; Anti-Patterns; Workflow Bottlenecks; Other Resources; Index; Colophon;