Synopses & Reviews
There's been a massive amount of innovation in data tools over the last few years, thanks to a few key trends:
Learning from the web. Techniques originally developed by website developers coping with scaling issues are increasingly being applied to other domains.
CS+?=$$$. Google have proven that research techniques from computer science can be effective at solving problems and creating value in many real-world situations. That's led to increased interest in cross-pollination and investment in academic research from commercial organizations.
Cheap hardware*. Now that machines with a decent amount of processing power can be hired for just a few cents an hour, many more people can afford to do large-scale data processing. They can't afford the traditional high prices of professional data software though, so they've turned to open-source alternatives.
These trends have led to a Cambrian Explosion of new tools, which means when you're planning a new data project you have a lot to choose from. This guide aims to help you make those choices by describing each tool from the perspective of a developer looking to use them in an application. Wherever possible, this will be from my first-hand experiences, or from colleagues who have used the systems in production environments. I've made a deliberate choice to include my own opinions and impressions, so you should see this guide as a starting point for exploring the tools, not the final word. I'll do my best to explain what I like about each service but your tastes and requirements may well be quite different.
Since the goal is to help experienced engineers navigate the new data landscape, the guide only covers tools that have been created or risen to prominence in the last few years. For example, PostGres is not covered because it's been widely used for over a decade, but its Greenplum derivative is newer and less well-known, so it is included.
To help you navigate the large number of new data tools available, this guide describes 60 of the most recent innovations, from NoSQL databases and MapReduce approaches to machine learning and visualization tools. Descriptions are based on first-hand experience with these tools in a production environment.
This handy glossary also includes a chapter of key terms that help define many of these tool categories:
- NoSQL Databases—Document-oriented databases using a key/value interface rather than SQL
- MapReduce—Tools that support distributed computing on large datasets
- Storage—Technologies for storing data in a distributed way
- Servers—Ways to rent computing power on remote machines
- Processing—Tools for extracting valuable information from large datasets
- Natural Language Processing—Methods for extracting information from human-created text
- Machine Learning—Tools that automatically perform data analyses, based on results of a one-off analysis
- Visualization—Applications that present meaningful data graphically
- Acquisition—Techniques for cleaning up messy public data sources
- Serialization—Methods to convert data structure or object state into a storable format
About the Author
A former Apple engineer, Pete Warden is the founder of OpenHeatMap, and writes on large-scale data processing and visualization.
Table of Contents
Preface; Conventions Used in This Book; Using Code Examples; Safari® Books Online; How to Contact Us; Chapter 1: Terms; 1.1 Document-Oriented; 1.2 Key/Value Stores; 1.3 Horizontal or Vertical Scaling; 1.4 MapReduce; 1.5 Sharding; Chapter 2: NoSQL Databases; 2.1 MongoDB; 2.2 CouchDB; 2.3 Cassandra; 2.4 Redis; 2.5 BigTable; 2.6 HBase; 2.7 Hypertable; 2.8 Voldemort; 2.9 Riak; 2.10 ZooKeeper; Chapter 3: MapReduce; 3.1 Hadoop; 3.2 Hive; 3.3 Pig; 3.4 Cascading; 3.5 Cascalog; 3.6 mrjob; 3.7 Caffeine; 3.8 S4; 3.9 MapR; 3.10 Acunu; 3.11 Flume; 3.12 Kafka; 3.13 Azkaban; 3.14 Oozie; 3.15 Greenplum; Chapter 4: Storage; 4.1 S3; 4.2 Hadoop Distributed File System; Chapter 5: Servers; 5.1 EC2; 5.2 Google App Engine; 5.3 Elastic Beanstalk; 5.4 Heroku; Chapter 6: Processing; 6.1 R; 6.2 Yahoo! Pipes; 6.3 Mechanical Turk; 6.4 Solr/Lucene; 6.5 ElasticSearch; 6.6 Datameer; 6.7 BigSheets; 6.8 Tinkerpop; Chapter 7: NLP; 7.1 Natural Language Toolkit; 7.2 OpenNLP; 7.3 Boilerpipe; 7.4 OpenCalais; Chapter 8: Machine Learning; 8.1 WEKA; 8.2 Mahout; 8.3 scikits.learn; Chapter 9: Visualization; 9.1 Gephi; 9.2 GraphViz; 9.3 Processing; 9.4 Protovis; 9.5 Fusion Tables; 9.6 Tableau; Chapter 10: Acquisition; 10.1 Google Refine; 10.2 Needlebase; 10.3 ScraperWiki; Chapter 11: Serialization; 11.1 JSON; 11.2 BSON; 11.3 Thrift; 11.4 Avro; 11.5 Protocol Buffers;