Synopses & Reviews
Discover how Apache Hadoop can unleash the power of your data. This comprehensive resource shows you how to build and maintain reliable, scalable, distributed systems with the Hadoop framework -- an open source implementation of MapReduce, the algorithm on which Google built its empire. Programmers will find details for analyzing datasets of any size, and administrators will learn how to set up and run Hadoop clusters.
This revised edition covers recent changes to Hadoop, including new features such as Hive, Sqoop, and Avro. It also provides illuminating case studies that illustrate how Hadoop is used to solve specific problems. Looking to get the most out of your data? This is your book.
- Use the Hadoop Distributed File System (HDFS) for storing large datasets, then run distributed computations over those datasets with MapReduce
- Become familiar with Hadoops data and I/O building blocks for compression, data integrity, serialization, and persistence
- Discover common pitfalls and advanced features for writing real-world MapReduce programs
- Design, build, and administer a dedicated Hadoop cluster, or run Hadoop in the cloud
- Use Pig, a high-level query language for large-scale data processing
- Analyze datasets with Hive, Hadoops data warehousing system
- Take advantage of HBase, Hadoops database for structured and semi-structured data
- Learn ZooKeeper, a toolkit of coordination primitives for building distributed systems
"Now you have the opportunity to learn about Hadoop from a master -- not only of the technology, but also of common sense and plain talk."
--Doug Cutting, Cloudera
If your organization is looking for a storage solution to accommodate a virtually endless amount of data, this book will show you how Apache HBase can fulfill your needs. As the open source implementation of Google's BigTable architecture, HBase scales to billions of rows and millions of columns, while ensuring that write and read performance remain constant. HBase: The Definitive Guide provides the details you require, whether you simply want to evaluate this high-performance, non-relational database, or put it into practice right away.
HBase's adoption rate is beginning to climb, and several IT executives are asking pointed questions about this high-capacity database. This is the only book available to give you meaningful answers.
- Learn how to distribute large datasets across an inexpensive cluster of commodity servers
- Develop HBase clients in many programming languages, including Java, Python, and Ruby
- Get details on HBase's primary storage system, HDFS—Hadoops distributed and replicated filesystem
- Learn how HBase's native interface to Hadoops MapReduce framework enables easy development and execution of batch jobs that can scan entire tables
- Discover the integration between HBase and other facets of the Apache Hadoop project
Hadoop in Action teaches readers how to use Hadoop and write MapReduce programs. The intended readers are programmers, architects, and project managers who have to process large amounts of data offline. Hadoop in Action will lead the reader from obtaining a copy of Hadoop to setting it up in a cluster and writing data analytic programs.
The book begins by making the basic idea of Hadoop and MapReduce easier to grasp by applying the default Hadoop installation to a few easy-to-follow tasks, such as analyzing changes in word frequency across a body of documents. The book continues through the basic concepts of MapReduce applications developed using Hadoop, including a close look at framework components, use of Hadoop for a variety of data analysis tasks, and numerous examples of Hadoop in action.
Hadoop in Action will explain how to use Hadoop and present design patterns and practices of programming MapReduce. MapReduce is a complex idea both conceptually and in its implementation, and Hadoop users are challenged to learn all the knobs and levers for running Hadoop. This book takes you beyond the mechanics of running Hadoop, teaching you to write meaningful programs in a MapReduce framework.
This book assumes the reader will have a basic familiarity with Java, as most code examples will be written in Java. Familiarity with basic statistical concepts (e.g. histogram, correlation) will help the reader appreciate the more advanced data processing examples.
'\'Organizations large and small are adopting Apache Hadoop to deal with huge application data sets, and this comprehensive resource provides the key for unlocking the wealth this data holds.\\n
If you're looking for a scalable storage solution to accommodate a virtually endless amount of data, this book shows you how Apache HBase can fulfill your needs. As the open source implementation of Google's BigTable architecture, HBase scales to billions of rows and millions of columns, while ensuring that write and read performance remain constant. Many IT executives are asking pointed questions about HBase. This book provides meaningful answers, whether youre evaluating this non-relational database or planning to put it into practice right away.
- Discover how tight integration with Hadoop makes scalability with HBase easier
- Distribute large datasets across an inexpensive cluster of commodity servers
- Access HBase with native Java clients, or with gateway servers providing REST, Avro, or Thrift APIs
- Get details on HBases architecture, including the storage format, write-ahead log, background processes, and more
- Integrate HBase with Hadoop's MapReduce framework for massively parallelized data processing jobs
- Learn how to tune clusters, design schemas, copy tables, import bulk data, decommission nodes, and many other tasks
About the Author
Tom White has been an Apache Hadoop committer since February 2007, and is a member of the Apache Software Foundation. He works for Cloudera, a company set up to offer Hadoop support and training. Previously he was as an independent Hadoop consultant, working with companies to set up, use, and extend Hadoop. He has written numerous articles for O'Reilly, java.net and IBM's developerWorks, and has spoken at several conferences, including at ApacheCon 2008 on Hadoop. Tom has a Bachelor's degree in Mathematics from the University of Cambridge and a Master's in Philosophy of Science from the University of Leeds, UK.
Table of Contents
Foreword; Preface; Administrative Notes; Whats in This Book?; Whats New in the Second Edition?; Conventions Used in This Book; Using Code Examples; Safari® Books Online; How to Contact Us; Acknowledgments; Chapter 1: Meet Hadoop; 1.1 Data!; 1.2 Data Storage and Analysis; 1.3 Comparison with Other Systems; 1.4 A Brief History of Hadoop; 1.5 Apache Hadoop and the Hadoop Ecosystem; Chapter 2: MapReduce; 2.1 A Weather Dataset; 2.2 Analyzing the Data with Unix Tools; 2.3 Analyzing the Data with Hadoop; 2.4 Scaling Out; 2.5 Hadoop Streaming; 2.6 Hadoop Pipes; Chapter 3: The Hadoop Distributed Filesystem; 3.1 The Design of HDFS; 3.2 HDFS Concepts; 3.3 The Command-Line Interface; 3.4 Hadoop Filesystems; 3.5 The Java Interface; 3.6 Data Flow; 3.7 Parallel Copying with distcp; 3.8 Hadoop Archives; Chapter 4: Hadoop I/O; 4.1 Data Integrity; 4.2 Compression; 4.3 Serialization; 4.4 File-Based Data Structures; Chapter 5: Developing a MapReduce Application; 5.1 The Configuration API; 5.2 Configuring the Development Environment; 5.3 Writing a Unit Test; 5.4 Running Locally on Test Data; 5.5 Running on a Cluster; 5.6 Tuning a Job; 5.7 MapReduce Workflows; Chapter 6: How MapReduce Works; 6.1 Anatomy of a MapReduce Job Run; 6.2 Failures; 6.3 Job Scheduling; 6.4 Shuffle and Sort; 6.5 Task Execution; Chapter 7: MapReduce Types and Formats; 7.1 MapReduce Types; 7.2 Input Formats; 7.3 Output Formats; Chapter 8: MapReduce Features; 8.1 Counters; 8.2 Sorting; 8.3 Joins; 8.4 Side Data Distribution; 8.5 MapReduce Library Classes; Chapter 9: Setting Up a Hadoop Cluster; 9.1 Cluster Specification; 9.2 Cluster Setup and Installation; 9.3 SSH Configuration; 9.4 Hadoop Configuration; 9.5 Security; 9.6 Benchmarking a Hadoop Cluster; 9.7 Hadoop in the Cloud; Chapter 10: Administering Hadoop; 10.1 HDFS; 10.2 Monitoring; 10.3 Maintenance; Chapter 11: Pig; 11.1 Installing and Running Pig; 11.2 An Example; 11.3 Comparison with Databases; 11.4 Pig Latin; 11.5 User-Defined Functions; 11.6 Data Processing Operators; 11.7 Pig in Practice; Chapter 12: Hive; 12.1 Installing Hive; 12.2 An Example; 12.3 Running Hive; 12.4 Comparison with Traditional Databases; 12.5 HiveQL; 12.6 Tables; 12.7 Querying Data; 12.8 User-Defined Functions; Chapter 13: HBase; 13.1 HBasics; 13.2 Concepts; 13.3 Installation; 13.4 Clients; 13.5 Example; 13.6 HBase Versus RDBMS; 13.7 Praxis; Chapter 14: ZooKeeper; 14.1 Installing and Running ZooKeeper; 14.2 An Example; 14.3 The ZooKeeper Service; 14.4 Building Applications with ZooKeeper; 14.5 ZooKeeper in Production; Chapter 15: Sqoop; 15.1 Getting Sqoop; 15.2 A Sample Import; 15.3 Generated Code; 15.4 Database Imports: A Deeper Look; 15.5 Working with Imported Data; 15.6 Importing Large Objects; 15.7 Performing an Export; 15.8 Exports: A Deeper Look; Chapter 16: Case Studies; 16.1 Hadoop Usage at Last.fm; 16.2 Hadoop and Hive at Facebook; 16.3 Nutch Search Engine; 16.4 Log Processing at Rackspace; 16.5 Cascading; 16.6 TeraByte Sort on Apache Hadoop; 16.7 Using Pig and Wukong to Explore Billion-edge Network Graphs; Installing Apache Hadoop; Prerequisites; Installation; Configuration; Clouderas Distribution for Hadoop; Preparing the NCDC Weather Data; Colophon;