Presto: Interacting with petabytes of data at Facebook

By Lydia Chan on Wednesday, November 6, 2013 at 7:01pm

By Martin Traverso
Background
Facebook is a data-driven company. Data processing and analytics are at the heart of building and delivering products for the 1 billion+active users of Facebook. We have one of the largest data warehouses in the world, storing more than 300 petabytes. The data is used for a wide range of applications, from traditional batch processing to graph analytics [1], machine learning, and real-time interactive analytics.
For the analysts, data scientists, and engineers who crunch data,derive insights, and work to continuously improve our products, the performance of queries against our data warehouse is important. Being able to run more queries and get results faster improves their productivity.
Facebook's warehouse data is stored in a few large Hadoop/HDFS-based clusters. Hadoop MapReduce [2] and Hive are designed for large-scale, reliable computation, and are optimized for overall system throughput. But as our warehouse grew to petabyte scale and our needs evolved, it became clear that we needed an interactive system optimized for low query latency.
In Fall 2012, a small team in the Facebook Data Infrastructure group set out to solve this problem for our warehouse users. We evaluated a few external projects, but they were either too nascent or did not meet our requirements for flexibility and scale. So we decided to build Presto, a new interactive query system that could operate fast at petabyte scale.
In this post, we will briefly describe the architecture of Presto, its current status, and future roadmap.
Architecture
Presto is a distributed SQL query engine optimized for ad-hoc analysis at interactive speed. It supports standard ANSI SQL, including complex queries, aggregations, joins, and window functions.
The diagram below shows the simplified system architecture of Presto. The client sends SQL to the Presto coordinator. The coordinator parses, analyzes, and plans the query execution. The scheduler wires together the execution pipeline, assigns work to nodes closest to the data, and monitors progress. The client pulls data from output stage, which in turn pulls data from underlying stages.
The execution model of Presto is fundamentally different from Hive/MapReduce. Hive translates queries into multiple stages of MapReduce tasks that execute one after another. Each task reads inputs from disk and writes intermediate output back to disk. In contrast, the Presto engine does not use MapReduce. It employs a custom query and execution engine with operators designed to support SQL semantics. In addition to improved scheduling, all processing is in memory and pipelined across the network between stages. This avoids unnecessary I/O and associated latency overhead. The pipelined execution model runs multiple stages at once, and streams data from one stage to the next as it becomes available. This significantly reduces end-to-end latency for many types of queries.

The Presto system is implemented in Java because it's fast to develop, has a great ecosystem, and is easy to integrate with the rest of the data infrastructure components at Facebook that are primarily built in Java. Presto dynamically compiles certain portions of the query plan down to byte code which lets the JVM optimize and generate native machine code. Through careful use of memory and data structures, Presto avoids typical issues of Java code related to memory allocation and garbage collection. (In a later post, we will share some tips and tricks for writing high-performance Java system code and the lessons learned while building Presto.)
Extensibility is another key design point for Presto. During the initial phase of the project, we realized that large data sets were being stored in many other systems in addition to HDFS. Some data stores are well-known systems such as HBase, but others are custom systems such as the Facebook News Feed backend. Presto was designed with a simple storage abstraction that makes it easy to provide SQL query capability against these disparate data sources. Storage plugins (called connectors) only need to provide interfaces for fetching metadata, getting data locations, and accessing the data itself. In addition to the primary Hive/HDFS backend, we have built Presto connectors to several other systems, including HBase, Scribe, and other custom systems.

Current status
As mentioned above, development on Presto started in Fall 2012. We had our first production system up and running in early 2013. It was fully rolled out to the entire company by Spring 2013. Since then, Presto has become a major interactive system for the company's data warehouse. It is deployed in multiple geographical regions and we have successfully scaled a single cluster to 1,000 nodes. The system is actively used by over a thousand employees,who run more than 30,000 queries processing one petabyte daily.
Presto is 10x better than Hive/MapReduce in terms of CPU efficiency and latency for most queries at Facebook. It currently supports a large subset of ANSI SQL, including joins, left/right outer joins, subqueries,and most of the common aggregate and scalar functions, including approximate distinct counts (using HyperLogLog) and approximate percentiles (based on quantile digest). The main restrictions at this stage are a size limitation on the join tables and cardinality of unique keys/groups. The system also lacks the ability to write output data back to tables (currently query results are streamed to the client).
Roadmap
We are actively working on extending Presto functionality and improving performance.  In the next few months, we will remove restrictions on join and aggregation sizes and introduce the ability to write output tables.  We are also working on a query "accelerator" by designing a new data format that is optimized for query processing and avoids unnecessary transformations. This feature will allow hot subsets of data to be cached from backend data store, and the system will transparently use cached data to "accelerate" queries.  We are also working on a high performance HBase connector.
Open source
After our initial Presto announcement at the Analytics @ WebScale conference in June 2013 [3], there has been a lot of interest from the external community. In the last couple of months, we have released Presto code and binaries to a small number of external companies. They have successfully deployed and tested it within their environments and given us great feedback.
Today we are very happy to announce that we are open-sourcing Presto. You can check out the code and documentation on the site below. We look forward to hearing about your use cases and how Presto can help with your interactive analysis.
http://prestodb.io/
https://github.com/facebook/presto
The Presto team within Facebook Data Infrastructure consists of Martin Traverso, Dain Sundstrom, David Phillips, Eric Hwang, Nileema Shingte and Ravi Murthy.
Links
[1] Scaling Apache Giraph to a trillion edges. https://www.facebook.com/notes/facebook-engineering/scaling-apache-giraph-to-a-trillion-edges/10151617006153920
[2] Under the hood: Scheduling MapReduce jobs more efficiently with Corona https://www.facebook.com/notes/facebook-engineering/under-the-hood-scheduling-mapreduce-jobs-more-efficiently-with-corona/10151142560538920
[3] Video of Presto talk at Analytics@Webscale conference, June 2013 https://www.facebook.com/photo.php?v=10202463462128185

Sent from Evernote

SQL is what’s next for Hadoop: Here’s who’s doing it

SQL is what’s next for Hadoop: Here’s who’s doing it:
When we first began putting together the schedule for Structure: Data several months ago, we knew that running SQL queries on Hadoop would be a big deal — we just didn’t know how big a deal it would actually become. Fast-forward to today, a mere month away from the event (March 20-21 in New York), and the writing on the wall is a lot clearer. SQL support isn’t the end-game for Hadoop, but it’s the feature that will help Hadoop find its way into more places in more companies that understand the importance of next-generation analytics but don’t want to (or can’t yet) re-invent the wheel by becoming MapReduce experts.
In fact, there are now so many products and projects pushing SQL queries and interactive data analysis on Hadoop — including two more announced this week — that it’s getting hard to keep track. But I’ll do my best.
Of course, Facebook began this whole movement to bring SQL database-like functionality to Hadoop when it created Hive in 2009. Hive, now an Apache project, includes a data-management layer and SQL-like query language called HiveQL. It has proven rather useful and popular over the years, but Hive’s reliance on MapReduce makes it somewhat slow by nature — MapReduce scans the entire data set and moves a lot of data over the network while processing a job — and there hasn’t been much effort to package it in a manner that might attract mainstream users.
And keep in mind that this next generation of SQL-on-Hadoop tools aren’t just business intelligence or database products that can access data stored in Hadoop; EMC Greenplum, HP Vertica, IBM Netezza, ParAccel, Microsoft SQL Server and Teradata/Aster Data (which this week released some cool new features for just this purpose) all allow some sort of access to Hadoop data. Rather, these are applications, frameworks and engines that let users query Hadoop data from inside Hadoop, sometimes by re-architecting the underlying compute and data infrastructures. The beauty of this approach is that data is usable in its existing form and, in theory, doesn’t require two separate data stores for analytic applications.

Data warehouses and BI: The Structure: Data set

I’m highlighting this group of companies first, not because I think they’re the best (although that might well be), but because I’m truly excited about the panel they’ll be featured on at our conference next month. The panel is moderated by Facebook engineering manager Ravi Murthy– a guy who knows his way around a database — so they’ll have to answer some tough questions from one of the most-advanced and most-aggressive Hadoop and analytics tools users out there:
Apache Drill: Drill is a MapR-led effort to create a Google Dremel-like (or BigQuery-like) interactive query engine on top of Hadoop. First announced in August, the project is still under development and in the incubator program within Apache. According to its web site, “One explicitly stated design goal is that Drill is able to scale to 10,000 servers or more and to be able to process petabyes of data and trillions of records in seconds.”
Hadapt: Hadapt, which actually launched at Structure: Data in 2011, was the first of the SQL on Hadoop vendors and is somewhat unique in that it has a real product on the market and real users in production. Its unique architecture includes tools for advanced SQL functions and a split-execution engine for MapReduce and relational tasks, and both HDFS and relational storage. In October, the company announced a tight integration with Tableau Software around advanced visual analytics.

Platfora: Technically not a SQL product, Platfora is red-hot right now and is trying to re-imagine the world of business intelligence for a big data world. Essentially an HTML5 canvas laid atop Hadoop and an in-memory, massively parallel processing engine, the company’s software, which it unveiled in October, is designed to make analyzing data stored in Hadoop a fast and visually intuitive process.
Qubole: Qubole is an interesting case in that it’s essentially a cloud-based version of the popular Apache Hive framework launched by the guys who created Hive while working at Facebook. Qubole claims it auto-scaling abilities, optimized Hadoop code and columnar data cache make its service run much faster than Hive alone — and running on Amazon Web Services makes it easier than maintaining a physical cluster.

Data warehouses and BI: The rest

Citus Data: Citus Data’s CitusDB isn’t just about Hadoop, but rather wants to bring the power of its distributed Postgres implementation to all types of data. It relies on Postgres’s foreign data wrappers feature to convert disparate data types into the database’s native format, and then on its own distributed-processing technology to carry out queries in seconds or less. Because of its Postgres foundation, CitusDB can join data from different data sources and retains all the native features that come with that database.

Cloudera Impala:  Cloudera’s Impala might just be the most-important SQL-on-Hadoop effort around because of Cloudera’s expansive installation and partner footprints. It’s a massively parallel processing engine that bypasses MapReduce to enable interactive queries on data stored in either HDFS or HBase, using the same variant of SQL that Hive uses. However, because Cloudera doesn’t build applications, it’s relying on higher-level BI and analytics partners to provide the user interface.

Karmasphere: Karmasphere is one of the first startups to build an analytic application atop Hadoop, and in its 2.0 release last year the company added support for SQL queries of data in HDFS. Like Hive, Karmasphere still relies on MapReduce to process queries, which means it’s inherently slower than newer approaches. However, unlike Hive, Karmasphere allows for parallel queries to run at the same time and includes a visual interface for writing queries and filtering results.

Lingual: Lingual is a new open source project from Concurrent (see disclosure), the parent company of the Cascading framework for Hadoop. Announced on Wednesday, Lingual runs on Cascading and gives developers and analysts a true ANSI SQL interface from which to run analytics or build applications. Lingual is compatible with traditional BI tools, JDBC  and the Cascading family of APIs.
Phoenix: Phoenix is a new and relatively unknown open source project that comes out of Salesforce.com and aims to allow fast SQL queries of data stored in HBase, the NoSQL database built atop HDFS. Its stated mission: “Become the standard means of accessing HBase data through a well-defined, industry standard API.” Users interact with it through JDBC interfaces, and its developers claim its sub-second response times for small queries and seconds-long response for querying tens of millions of rows.

A sample of Phoenix via the SQuirreL client

Shark: Shark isn’t technically Hadoop, but it’s cut from the same cloth. Shark, in this case, stands for “Hive on Spark,” with Hive meaning the same thing it does to Hadoop, but with Spark being an in-memory platform designed to run parallel-processing jobs 100 times faster than MapReduce (a speed improve over traditional Hive that Shark also claims). Shark also includes APIs for turning query results into a type of data format amenable to machine learning algorithms. Both Shark and Spark are developed by the University of California, Berkeley’s AMPLab.

Stinger Initiative: Launched on Wednesday (along with a security gateway called Knox and a faster, simpler processing framework called Tez), the Stinger Initiative is a Hortonworks-led effort to make Hive faster — up too 100x — and more functional. Stinger adds more SQL analytics capabilities to Hive, but the most-important aspects are infrastructural: an optimized execution engine, a columnar file format and the ability to avoid MapReduce bottlenecks by running atop Tez.

Operational SQL

Drawn to Scale: Drawn to Scale is a startup that has built an operational SQL database on top of HBase. The key word here is database, as its product, called Spire, is modeled after Google’s F1 designed to power transactional applications as analytic ones. Spire has a fully distributed index and queries are sent only to the node with the relevant data, so reads and writes are fast and the system can handle lots of concurrent users without falling down.

Splice Machine: Database startup Splice Machine is also trying to get into the operational space by building its Splice SQL Engine atop the naturally distributed HBase database. Splice Machine focuses its message on transactional integrity, which is really where it separates itself from scalable NoSQL databases and analytics-focused SQL-on-Hadoop efforts. It relies on HBase’s aut0-sharding feature in order to making scaling an easy process.

Feature image courtesy of Shutterstock user hauhu.
Upcoming: Structure:Data, Mar. 20-21, 2013, New York, Register by March 1 and save $200! More upcoming conferences.



Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Infrastructure Q1: Cloud and big data woo enterprises
• A near-term outlook for big data

Apache Mahout: Scalable machine learning and data mining

Apache Mahout: Scalable machine learning and data mining:

The Apache Mahout™ machine learning library's goal is to build scalable machine learning libraries.

Mahout currently has

• Collaborative Filtering
• User and Item based recommenders
• K-Means, Fuzzy K-Means clustering
• Mean Shift clustering
• Dirichlet process clustering
• Latent Dirichlet Allocation
• Singular value decomposition
• Parallel Frequent Pattern mining
• Complementary Naive Bayes classifier
• Random forest decision tree based classifier
• High performance java collections (previously colt collections)
• A vibrant community
• and many more cool stuff to come by this summer thanks to Google summer of code

With scalable we mean:

Scalable to reasonably large data sets. Our core algorithms for clustering, classfication and batch based collaborative filtering are implemented on top of Apache Hadoop using the map/reduce paradigm. However we do not restrict contributions to Hadoop based implementations: Contributions that run on a single node or on a non-Hadoop cluster are welcome as well. The core libraries are highly optimized to allow for good performance also for non-distributed algorithms

Scalable to support your business case. Mahout is distributed under a commercially friendly Apache Software license.

Scalable community. The goal of Mahout is to build a vibrant, responsive, diverse community to facilitate discussions not only on the project itself but also on potential use cases. Come to the mailing lists to find out more.

Currently Mahout supports mainly four use cases: Recommendation mining takes users' behavior and from that tries to find items users might like. Clustering takes e.g. text documents and groups them into groups of topically related documents. Classification learns from exisiting categorized documents what documents of a specific category look like and is able to assign unlabelled documents to the (hopefully) correct category. Frequent itemset mining takes a set of item groups (terms in a query session, shopping cart content) and identifies, which individual items usually appear together.

Tutorial: Designing Good Algorithms for Map-Reduce and Beyond

Program - ACM Symposium on Cloud Computing 2012:

Tutorial: Designing Good Algorithms for Map-Reduce and Beyond (view)

Foto N. Afrati (NTUA and Google), Magdalena Balazinska (University of Washington), Anish Das Sarma (Google), Bill Howe (University of Washington), Semih Salihoglu (Stanford University), and Jeffrey D. Ullman (Stanford University)

• Slide set 1 (view)
• Slide set 2 (view)
• Slide set 3 (view)
• Slide set 4 (view)
• Slide set 5 (view)
• Slide set 6 (view)
• Slide set 8 (view)
• Slide set 9 (view)
• Slide set 10 (view)
• Slide set 11 (view)
• Slide set 12 (view)
• Slide set 13 (view)

Statistics about Hadoop and Mapreduce Algorithm Papers

Statistics about Hadoop and Mapreduce Algorithm Papers:

Underneath are statistics about which 20 papers (of about 80 papers) were most read in our 3 previous postings about mapreduce and hadoop algorithms (the postings have been read approximately 5000 times). The list is ordered by decreasing reading frequency, i.e. most popular at spot 1.

1. MapReduce-Based Pattern Finding Algorithm Applied in Motif Detection for Prescription Compatibility Network
   authors: Yang Liu, Xiaohong Jiang, Huajun Chen , Jun Ma and Xiangyu Zhang – Zhejiang University
2. Data-intensive text processing with Mapreduce
   authors: Jimmy Lin and Chris Dyer – University of Maryland
3. Large-Scale Behavioral Targeting
   authors: Ye Chen (eBay), Dmitry Pavlov (Yandex Labs) and John F. Canny (University of California, Berkeley)
4. Improving Ad Relevance in Sponsored Search
   authors: Dustin Hillard, Stefan Schroedl, Eren Manavoglu, Hema Raghavan and Chris Leggetter (Yahoo Labs)
5. Experiences on Processing Spatial Data with MapReduce
   authors: Ariel Cary, Zhengguo Sun, Vagelis Hristidis and Naphtali Rishe – Florida International University
6. Extracting user profiles from large scale data
   authors: Michal Shmueli-Scheuer, Haggai Roitman, David Carmel, Yosi Mass and David Konopnicki – IBM Research, Haifa
7. Predicting the Click-Through Rate for Rare/New Ads
   authors: Kushal Dave and Vasudeva Varma – IIIT Hyderabad
8. Parallel K-Means Clustering Based on MapReduce
   authors: Weizhong Zhao, Huifang Ma and Qing He – Chinese Academy of Sciences
9. Storage and Retrieval of Large RDF Graph Using Hadoop and MapReduce
   authors: Mohammad Farhan Husain, Pankil Doshi, Latifur Khan and Bhavani Thuraisingham – University of Texas at Dallas
10. Map-Reduce Meets Wider Varieties of Applications
    authors: Shimin Chen and Steven W. Schlosser – Intel Research
11. LogMaster: Mining Event Correlations in Logs of Large-scale Cluster Systems
    authors: Wei Zhou, Jianfeng Zhan, Dan Meng (Chinese Academy of Sciences), Dongyan Xu (Purdue University) and Zhihong Zhang (China Mobile Research)
12. Efficient Clustering of Web-Derived Data Sets
    authors: Luıs Sarmento, Eugenio Oliveira (University of Porto), Alexander P. Kehlenbeck (Google), Lyle Ungar (University of Pennsylvania)
13. A novel approach to multiple sequence alignment using hadoop data grids
    authors: G. Sudha Sadasivam and G. Baktavatchalam – PSG College of Technology
14. Web-Scale Distributional Similarity and Entity Set Expansion
    authors: Patrick Pantel, Eric Crestan, Ana-Maria Popescu, Vishnu Vyas (Yahoo Labs) and Arkady Borkovsky (Yandex Labs)
15. Grammar based statistical MT on Hadoop
    authors: Ashish Venugopal and Andreas Zollmann (Carnegie Mellon University)
16. Distributed Algorithms for Topic Models
    authors: David Newman, Arthur Asuncion, Padhraic Smyth and Max Welling – University of California, Irvine
17. Parallel algorithms for mining large-scale rich-media data
    authors: Edward Y. Chang, Hongjie Bai and Kaihua Zhu – Google Research
18. Learning Influence Probabilities In Social Networks
    authors: Amit Goyal, Laks V. S. Lakshmanan (University of British Columbia) and Francesco Bonchi (Yahoo! Research)
19. MrsRF: an efficient MapReduce algorithm for analyzing large collections of evolutionary trees
    authors: Suzanne J Matthews and Tiffani L Williams – Texas A&M University
20. User-Based Collaborative-Filtering Recommendation Algorithms on Hadoop
    authors: Zhi-Dan Zhao and Ming-sheng Shang

Hadoop's tremendous inefficiency on graph data management (and how to avoid it)

Hadoop's tremendous inefficiency on graph data management (and how to avoid it): "Hadoop is great. It seems clear that it will serve as the basis of the vast majority of analytical data management within five years. Already today it is extremely popular for unstructured and polystructured data analysis and processing, since it is hard to find other options that are superior from a price/performance perspective. The reader should not take the following as me blasting Hadoop. I believe that Hadoop (with its ecosystem) is going to take over the world.

The problem with Hadoop is that its strength is also its weakness. Hadoop gives the user tremendous flexibility and power to scale all kinds of different data management problems. This is obviously great. But it is this same flexibility that allows the user to perform incredibly inefficient things and not care because (a) they can simply add more machines and use Hadoop's scalability to hide inefficiency in user code (b) they can convince themselves that since everyone talks about Hadoop as being designed for 'batch data processing' anyways, they can let their process run in the background and not care about how long it will take for it to return.

Although not the subject of this post, an example of this inefficiency can be found in a SIGMOD paper that a bunch of us from Yale and the University of Wisconsin published 5 weeks ago. The paper shows that using Hadoop on structured (relational) data is at least a factor of 50 less efficient than it needs to be (an incredibly large number given how hard data center administrators work to yield less than a factor of two improvement in efficiency). As many readers of this blog already know, this factor of 50 improvement is the reason why Hadapt was founded. But this post is not about Hadapt or relational data. In this post, the focus is on graph data, and how if one is not careful, using Hadoop can be well over a factor of 1000 less efficient than it needs to be.

Before we get into how to improve Hadoop's efficiency on graph data by a factor of 1000, let's pause for a second to comprehend how dangerous it is to let inefficiencies in Hadoop become widespread. Imagine a world where the vast majority of data processing runs on Hadoop (a not entirely implausible scenario). If people allow these factors of 50 or 1000 to exist in their Hadoop utilization, these inefficiency factors translate directly to factors of 50 or 1000 more power utilization, more carbon emissions, more data center space, and more silicon waste. The disastrous environmental consequences in a world where everyone standardizes on incredibly inefficient technology is downright terrifying. And this is ignoring the impact on businesses in terms of server and energy costs, and lower performance. It seems clear that developing a series of 'best practices' around using Hadoop efficiently is going to be extremely important moving forward.

Let's delve into the subject of graph data in more detail. Recently there was a paper by Rohloff et. al. that showed how to store graph data (represented in vertex-edge-vertex 'triple' format) in Hadoop, and perform sub-graph pattern matching in a scalable fashion over this graph of data. The particular focus of the paper is on Semantic Web graphs (where the data is stored in RDF and the queries are performed in SPARQL), but the techniques presented in the paper are generalizable to other types of graphs. This paper and resulting system (called SHARD) has received significant publicity, including a presentation at HadoopWorld 2010, a presentation at DIDC 2011, and a feature on Cloudera's Website. In fact, it is a very nice technique. It leverages Hadoop to scale sub-graph pattern matching (something that has historically be difficult to do); and by aggregating all outgoing edges for a given vertex into the same key-value pair in Hadoop, it even scales queries in a way that is 2-3 times more efficient than the naive way to use Hadoop for the same task.

The only problem is that, as shown by an upcoming VLDB paper that we're releasing today, this technique is an astonishing factor of 1340 times less efficient than an alternative technique for processing sub-graph pattern matching queries within a Hadoop-based system that we introduce in our paper. Our paper, led by my student, Jiewen Huang, achieves these enormous speedups in the following ways:

1. Hadoop, by default, hash partitions data across nodes. In practice (e.g., in the SHARD paper) this results in data for each vertex in the graph being randomly distributed across the cluster (dependent on the result of a hash function applied to the vertex identifier). Therefore, data that is close to each other in the graph can end up very far away from each other in the cluster, spread out across many different physical machines. For graph operations such as sub-graph pattern matching, this is wildly suboptimal. For these types of operations, the graph is traversed by passing through neighbors of vertexes; it is hugely beneficial if these neighbors are stored physically near each other (ideally on the same physical machine). When using hash partitioning, since there is no connection between graph locality and physical locality, a large amount of network traffic is required for each hop in the query pattern being matched (on the order of one MapReduce job per graph hop), which results in severe inefficiency. Using a clustering algorithm to graph partition data across nodes in the Hadoop cluster (instead of using hash partitioning) is a big win.
   
   
2. Hadoop, by default, has a very simple replication algorithm, where all data is generally replicated a fixed number of times (e.g. 3 times) across the cluster. Treating all data equally when it comes to replication is quite inefficient. If data is graph partitioned across a cluster, the data that is on the border of any particular partition is far more important to replicate than the data that is internal to a partition and already has all of its neighbors stored locally. This is because vertexes that are on the border of a partition might have several of their neighbors stored on different physical machines. For the same reasons why it is a good idea to graph partition data to keep graph neighbors local, it is a good idea to replicate data on the edges of partitions so that vertexes are stored on the same physical machine as their neighbors. Hence, allowing different data to be replicated at different factors can further improve system efficiency.
   
   
3. Hadoop, by default, stores data on a distributed file system (HDFS) or a sparse NoSQL store (HBase). Neither of these data stores are optimized for graph data. HDFS is optimized for unstructured data, and HBase for semi-structured data. But there has been significant research in the database community on creating optimized data stores for graph-structured data. Using a suboptimal store for the graph data is another source of tremendous inefficiency. By replacing the physical storage system with graph-optimized storage, but keeping the rest of the system intact (similar to the theme of the HadoopDB project), it is possible to greatly increase the efficiency of the system.

To a first degree of approximation, each of the above three improvements yield an entire order of magnitude speedup (a factor of 10). By combining them, we therefore saw the factor of 1340 improvement in performance on the identical benchmark that was run in the SHARD paper. (For more details on the system architecture, partitioning and data placement algorithms, query processing, and experimental results please see our paper).

It is important to note that since we wanted to run the same benchmark as the SHARD paper, we used the famous Lehigh University Benchmark (LUBM) for Semantic Web graph data and queries. Semantic Web sub-graph pattern matching queries tend to contain quite a lot of constants (especially on edge labels) relative to other types of graph queries. The next step for this project is to extend and benchmark the system on other graph applications (the types of graphs that people tend to use systems based on Google's Pregel project today).

In conclusion, it is perfectly acceptable to give up a little bit of efficiency for improved scalability when using Hadoop. However, once this decrease in efficiency starts to reach a factor of two, it is likely a good idea to think about what is causing this inefficiency, and attempt to find ways to avoid it (while keeping the same scalability properties). Certainly once the factor extends beyond the factor of two (such as the enormous 1340 factor we discovered in our VLDB paper), the sheer waste in power and hardware cannot be ignored. This does not mean that Hadoop should be thrown away; however it will become necessary to package Hadoop with 'best practice' solutions to avoid such unnecessarily high levels of waste.

"

LexisNexis open-sources its Hadoop killer

LexisNexis open-sources its Hadoop killer: "

LexisNexis is releasing a set of open-source, data-processing tools that it says outperforms Hadoop and even handles workloads Hadoop presently can’t. The technology (and new business line) is called HPCC Systems, and was created 10 years ago within the LexisNexis Risk Solutions division that analyzes huge amounts of data for its customers in intelligence, financial services and other high-profile industries. There have been calls for a legitimate alternative to Hadoop, and this certainly looks like one.

According to Armando Escalante, CTO of LexisNexis Risk Solutions, the company decided to release HPCC now because it wanted to get the technology into the community before Hadoop became the de facto option for big data processing. Escalante told me during a phone call that he thinks of Hadoop as “a guy with a machete in front of a jungle — they made a trail,” but that he thinks HPCC is superior.

But in order to compete for mindshare and developers, he said, the company felt it had to open-source the technology. One big thing Hadoop has going for it is its open-source model, Escalante explained, which attracts a lot of developers and a lot of innovation. If his company wanted HPCC to “remain relevant” and keep improving through new use cases and ideas from a new community, the time for release was now and open source had to be the model.

Hadoop, of course, is the Apache Software Foundation project created several years ago by then-Yahoo employee Doug Cutting. It has become a critical tool for web companies — including Yahoo and Facebook — to process their ever-growing volumes of unstructured data, and is fast making its way into organizations of all types and sizes. Hadoop has spawned a number of commercial distributions and products, too, including from Cloudera, EMC  and IBM.

How HPCC works

Hadoop relies on two core components to store and process huge amounts of data: the Hadoop Distributed File System and Hadoop MapReduce. However, as Cloudant CEO Mike Miller explained in a post over the weekend, MapReduce is still a relatively complex language for writing parallel-processing workflows. HPCC seeks to remedy this with its Enterprise Control Language.

Escalante says ECL is a declarative, data-centric language that abstracts a lot of the work necessary within MapReduce. For certain tasks that take a thousand lines of code in MapReduce, he said, ECL only requires 99 lines. Furthermore, he explained, ECL doesn’t care how many nodes are in the cluster because the system automatically distributes data across however many nodes are present. Technically, though, HPCC could run on just a single virtual machine. And, says Escalante, HPCC is written in C++ — like the original Google MapReduce  on which Hadoop MapReduce is based — which he says makes it inherently faster than the Java-based Hadoop version.

HPCC offers two options for processing and serving data: the Thor Data Refinery Cluster and the Roxy Rapid Data Delivery Cluster. Escalante said Thor — so named for its hammer-like approach to solving the problem — crunches, analyzes and indexes huge amounts of data a la Hadoop. Roxie, on the other hand, is more like a traditional relational database or database warehouse that even can serve transactions to a web front end.

We didn’t go into detail on HPCC’s storage component, but Escalante noted that it does utilize a distributed file system, although it can support a variety of off-node storage architectures and/or local solid-state drives.

He added that in order to ensure LexisNexis wasn’t blinded by “eating its own dogfood,” his team hired a Hadoop expert to kick the tires on HPCC. The consultant was impressed, Escalante said, but did note some shortcomings that the team addressed as it readied the technology for release. It also built a converter for migrating Hadoop applications written in the Pig language to ECL.

Can HPCC Systems actually compete?

The million-dollar question is whether HPCC Systems can actually attract an ecosystem of contributors and users that will help it rise above the status of big data also-ran. Escalante thinks it can, in large part because HPCC already has been proven in production dealing with LexisNexis Risk Solutions’ 35,000 data sources, 5,000 transactions per second and large, paying customers. He added that the company also will provide enterprise licenses and proprietary applications in addition to the open-source code. Plus, it already has potential customers lined up.

It’s often said that competition means validation. Hadoop has moved from a niche set of tools to the core of a potentially huge business that’s growing every day, and even Microsoft has a horse in this race with its Dryad set of big data tools. Hadoop has already proven itself, but the companies and organizations relying on it for the their big data strategies can’t rest on their laurels.

Image courtesy of Flickr user NileGuide.com.

Related content from GigaOM Pro (subscription req’d):

• The Case for Increased M&A in 2011: Actions and Outlooks
• The Structure 50: The Top 50 Cloud Innovators
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

---

"

EMC Makes a Big Bet on Hadoop

EMC Makes a Big Bet on Hadoop: "

EMC is throwing its weight behind Hadoop. Today, at EMC World, the storage giant announced a slew of Hadoop-centric products, including a specialized appliance for Hadoop-based big data analytics and two separate Hadoop distributions. EMC’s entry is most definitely going to shake up the Hadoop and database markets. EMC is now the largest company actively pushing its own Hadoop distribution, and it has an appliance that will put EMC out in front of analytics vendors such as Oracle and Teradata when it comes to handling unstructured data.

EMC’s flagship Hadoop distribution is called Greenplum HD Enterprise Edition. EMC describes it as “a 100 percent interface-compatible implementation of the Apache Hadoop stack” that also includes enterprise-grade features such as snapshots and wide-area replication, a native network file system, and integrated storage and cluster management capabilities. The company also claims performance improvements of two to fives times over the standard Apache Hadoop distribution.

Mapr Magic

It’s noteworthy that many of these capabilities are also available in startup MapR’s HDFS alternative, and that MapR CEO John Schroeder took the stage at a morning EMC World press conference announcing the news. EMC Greenplum’s Luke Lonergan wouldn’t confirm to me that EMC’s Enterprise Edition will use MapR as the primary storage engine, but it’s not too difficult to connect the dots.

However, while the Enterprise Edition is proprietary in part, the Greenplum HD Community Edition is fully open source and still makes big improvements over what’s currently available with the Apache version. In fact, Lonergan told me, Community Edition is based on Facebook’s optimized version of Hadoop. Like Cloudera’s distribution for Hadoop, Community Edition pre-integrates Hadoop MapReduce, Hadoop Distributed File System, HBase, Zookeeper and Hive, but it also includes fault tolerance for the NameNode in HDFS and the JobTracker node in Hadoop MapReduce. These improvements are underway within Apache thanks to Yahoo , but they’re not included in any official release yet.

Too Much Hadoop?

I asked a couple of weeks ago whether the Hadoop-distribution market could handle all the players it now hosts, and now that question is even more pressing. As Luke Lonergan put it during the press conference, EMC is an “8,000-pound elephant” in the Hadoop space, and that should make Cloudera, IBM, DataStax and (possibly) Yahoo shake seek higher ground.

For Cloudera, EMC is major threat because it competes directly against Cloudera’s open-source and proprietary products. It even has partnerships with a large number of business intelligence and other up-the-stack vendors, some of which already are Cloudera partners. These include Concurrent, CSC, Datameer, Informatica, Jaspersoft, Karmasphere, Microstrategy, Pentaho, SAS, SnapLogic, Talend, and VMware.

Oh, and Cloudera and Greenplum have an existing integration partnership. As Lonergan noted, “This definitely marks a change [in that relationship].” The two are now competitors, after all.

EMC vs Big Blue

IBM is still the largest company involved in selling Hadoop products, but it presently suffers from the problem of not having yet announced its official Hadoop distribution. EMC’s Hadoop distributions will be available later this quarter. I noted recently how EMC is following IBM’s lead in acquiring capabilities across the big data stack — from Hadoop to predictive analytics — and today’s news further proves how competitive the two storage heavyweights might become in the analytics space, too.

IBM isn’t the only big-name vendor that should be worried about EMC’s new Hadoop-heavy plans, though. The EMC Greenplum HD Data Computing Appliance should make appliance makers Oracle and Teradata, as well as analytic database vendors such as HP, ParAccel and others, quite nervous. The appliance is like the existing EMC Greenplum Data Computing Appliance, only it lets customers process Hadoop data within the same system as their Greenplum analytic database. Presently, most analytic databases and appliances integrate with Hadoop, but still suffer from the latency of having to send data over the network from Hadoop clusters to the database and back.

IBM already has integrated Hadoop with its other big data tools, including with InfoSphere BigInsights, Watson and Cognos Consumer Insight, and I have to believe a version of its Netezza analytics appliance with Hadoop co-processing will be on the way shortly, possibly in conjunction with its official Hadoop distribution release.

Lonergan also noted that EMC is working closely with VMware, of which EMC is the majority stockholder, on integrating EMC’s Hadoop products with VMware’s virtualization and cloud products, as well as its GemStone distributed database software.

There still will be opportunities for community collaboration among all the open source Hadoop distributions — Cloudera, DataStax Brisk and EMC Greenplum HD Community Edition — but we’ll see how willing they are to work together now that the competition has really heated up. All of a sudden, EMC looks like the strongest Hadoop company going, and everyone else needs to figure out in a hurry how they’ll counter today’s landscape-altering news.

Related content from GigaOM Pro (subscription req’d):

• GigaOM Pro: This Week in Infrastructure: Facebook’s Hadoop Agenda
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Putting Big Data to Work: Opportunities for Enterprises

---

"

The Rise of Hadoop: How many Hadoop-related solutions exist?

The Rise of Hadoop: How many Hadoop-related solutions exist?: "The Rise of Hadoop: How many Hadoop-related solutions exist?:

The CMSWire commented list of Hadoop-related solutions:

1. Apache Hadoop

2. Appistry CloudIQ Storage Hadoop Edition: a HDFS replacement improving on the single NameNode ( here).

   
   

   Shipping.

   
   

3. IBM Distribution of Apache Hadoop: Apache Hadoop, a 32-bit Linux version of the IBM SDK for Java 6 SR 8, and an easy-to-use installer that will install and configure both Hadoop (including SSH setup) and Java (here).

   
   

   Shipping, but in alphaWorks

   
   

4. IBM Global Parallel File System (GPFS): a high-performance shared-disk clustered file system developed by IBM (here).

   
   

   Shipping.

   
   

5. Cloudera’s Distribution including Apache Hadoop: Cloudera’s packaging for Hadoop and Hadoop toolkit (here).

   
   

   Shipping.

   
   

6. DataStax Brisk: using Apache Cassandra for Hadoop (and Hive) core services (here).

   
   

   Announced, but not released yet

   
   

7. Amazon Elastic MapReduce: Amazon hosted Hadoop framework running on the infrastructure of Amazon EC2 and Amazon S3 (here).

   Shipping.

   
   

8. Mapr: proprietary replacement for HDFS.

   
   

   Talked about

   
   

9. CloudStore: the former Kosmos open-source distributed filesystem (here).

   
   

   Shipping^[1]

   
   

10. Pervasive DataRush: parallel data processing optimization for Hadoop jobs (here).

    
    

    Shipping.

    
    

11. Cascading: query API and query Planner.

    
    

    Shipping.

    
    

12. Apache Hive: data warehouse on top of Hadoop.

    
    

    Shipping

    
    

13. Yahoo Pig: high-level data-flow language and execution framework for parallel computation.

    
    

    Shipping.

    
    

14. Hadapt: hybrid architecture combining relational databases and Hadoop (here).

    
    

    Announced.

    
    

Some others are in the Hadoop toolkit.

Instead of “shipping” another criteria that can be used is number of users and amount of processed data.

---

1. 
   

   Kosmos current release is 0.5 dating June, 2010
    ↩

   
   

Original title and link: The Rise of Hadoop: How many Hadoop-related solutions exist? (NoSQL databases © myNoSQL)

"

As Big Data Takes Off, the Hadoop Wars Begin

As Big Data Takes Off, the Hadoop Wars Begin: "
It turns out “big data” isn’t just a buzzword, but a legitimate concern for companies across the board. Their interest in the tools to take advantage of the opportunity for analysis of all this data has sparked a land grab among established vendors and startups alike. The action is centered around Hadoop, the flagship technology for storing and processing large amounts of unstructured data.

Since Yahoo open-sourced Hadoop a few years ago, the primary options for organizations wanting to take advantage of the product have been the open-source Apache Hadoop distribution, the Cloudera distribution of Hadoop ,and Amazon Web Services’ Elastic MapReduce service. That will change soon, as everyone from EMC and IBM to database startups like Hadapt and DataStax get into the business of selling Hadoop-based technologies and services.

So far, Cloudera, which provides commercial support for its open-source distribution, as well as its own proprietary Hadoop-cluster management software, has been the only company to truly capitalize on Hadoop financially. Arguably, its success is to blame for the stiff competition it’s about to face for companies’ Hadoop attention and dollars.

Too Many Distributions

Cloudera, a private company, hasn’t released any financial details, but Wednesday at Structure: Big Data, VP of Engineering Amr Adawallah mentioned during a panel that Cloudera has more than 80 customers running Hadoop in production, and the company does have technology partnerships across the data world, including a leading data warehouse, BI, and database vendors. Cloudera also has raised $36 million from investors since launching in 2009. It appears the other software companies have noticed all the activity around Cloudera and want in on some of the action.

IBM already has a Hadoop business that includes its own distribution it says is better suited for commercial users than the open-source Apache Hadoop distribution, though both IBM and Cloudera are based on the Apache distribution. IBM’s offering also provides an application called InfoSphere BigSheets, which hides the complexities of Hadoop underneath a variety of advanced analytics, BI and visualization tools. Based on a few sources I spoke with at Structure: Big Data, and after reading into an advertisement in the program for the conference , it looks EMC is getting into the game. The ad hints that EMC will announce a Hadoop product involving its new Greenplum database on May 9: The ad read, “05.09.11: EMC Greenplum. Apache Hadoop.” Also at the event, two independent sources suggested members of Yahoo’s Hadoop team will be spinning off their own separate business, and there is speculation this move is somehow tied into EMC’s Hadoop plans.

IBM isn’t to be taken lightly, nor is EMC on its own, but the latter turn of events would be a potentially market-changing situation given the Hadoop know-how within Yahoo, which has contributed the majority of the code now included in Apache Hadoop. During a panel at Structure: Big Data, Yahoo’s VP of Cloud Architecture Todd Papaioannou, quipped to Cloudera’s Awadallah that Yahoo will keep innovating on Hadoop and everyone could keep reselling it. Papaioannou declined to comment on the rumors of a Hadoop spinout, but did tell me via email, “I think Apache Hadoop will remain the go-to place to get access to new improvements and innovation in the core Hadoop platform. That’s exactly why we announced our ‘double down’ strategy and the work we are doing on the next generation of both Map Reduce and HDFS.”

Death by a Thousand Startups

It’s not only large vendors that Cloudera will have to fight off; its real threat is death by a thousand startups and ISVs. At Structure: Big Data, NoSQL startup DataStax announced its own open-source Hadoop distribution based on the NoSQL database Cassandra, which provides a replacement for the Hadoop Distributed File System (HDFS). DataStax says this gives users the ability to process data and feed it back to applications at extremely low latencies, which Cloudera can’t offer because Apache Hadoop environments currently reside on separate infrastructure from application servers and databases. Om wrote earlier about Mapr, a startup focused on improving the performance and reliability of the HDFS. Appistry is already addressing this with its own wholly-distributed HDFS alternative.

Launches weren’t over yet. Another database startup called Hadapt officially launched Wednesday with a product that melds the HDFS-based HBase database with traditional RDBMS capabilities. HBase is an Apache Hadoop subproject heavily used by Facebook, and included as part of Cloudera’s Hadoop distribution. And next Tuesday, high-performance computing pioneer Platform Computing — which has a presence in many large financial data centers and 10 of the top 20 Fortune companies — will be announcing an analytics offering that applies its current cluster- and grid-management capabilities to MapReduce workloads. As noted above, management tools are where Cloudera actually makes money selling software as opposed to services.

There are several commercial alternatives to Apache Hadoop MapReduce, as well. Pervasive Software’s DataRush software is designed for writing big data workflows and to take full advantage of multi-core processors. And Cascading, an open-source, data-processing API sits atop MapReduce. A startup called Concurrent offers commercial support and services for Cascading. Amazon Web Services offers a cloud-based Hadoop service called Elastic MapReduce, which spares users the cost of buying their own gear on which to run Hadoop workloads.

Confused? Here’s a round-up of currently available Hadoop distributions:

Full-on distributions

• Apache Hadoop
• Cloudera’s Distribution including Apache Hadoop (that’s the official name)
• IBM Distribution of Apache Hadoop
• DataStax Brisk
• Amazon Elastic MapReduce

HDFS alternatives

• Mapr
• Appistry CloudIQ Storage Hadoop Edition
• IBM Global Parallel File System (GPFS)
• CloudStore

Hadoop MapReduce alternatives

• Pervasive DataRush
• Cascading
• Hive (an Apache subproject, included in Cloudera’s distribution)
• Pig (a Yahoo-developed language, included in Cloudera’s distribution)

Cloudera Isn’t Flinching — Yet

Even with all this competition, however, it’s unclear whether Cloudera actually feels its iron grip on the commercial Hadoop world slipping away. CEO Mike Olson thinks a rich ecosystem of Hadoop companies is necessary if it’s to grow into a multi-billion-dollar business like he thinks it can, but he sees most of that activity taking place up the stack from the foundational distribution layer where Cloudera operates. He said via email, “I believe there’s an enormous opportunity for smart companies, and even open-source projects, to build a new generation of data analysis tools on top of that platform.”

His colleague Awadallah was slightly less politic in his response when asked specifically about the DataStax distribution, stating in a video interview with my colleague Stacey Higginbotham Wednesday that he thinks DataStax’s distribution is a “big mistake,” and he doesn’t think the company can yet back up its claims of Hadoop support. He added that a better alternative to trying to reinvent the wheel in terms of Hadoop support and stability would have been for DataStax to keep its focus on Cassandra partner with Cloudera on the Hadoop integration.

Watch this video for free on GigaOM

Cloudera has plenty of reason to be confident, actually. Among its ranks are Hadoop creator Doug Cutting and former Yahoo colleague Awadallah, as well as Chief Scientist Jeff Hammerbacher – who previously led Facebook’s massive data efforts — and Vertica vetertan Omer Trajman. Olson himself is the former CEO of SleepyCat Software, which distributed the open-source Berkeley DB database before Oracle bought the company in 2006. Or, as Adwallah put it, “[W]e have the muscle to be able to back up our words with execution.” Further, as long as Facebook and Yahoo continue contributing their webscale-driven — and proven — enhancements back to Apache Hadoop, Cloudera has plenty of fuel to feed its evolution. Facebook, for example, is responsible for the popular Hive query language that gives Hadoop users a SQL-like experience many prefer to MapReduce, and, as noted above, Yahoo is currently pushing for a next-generation architecture that addresses some known performance bottlenecks with Apache Hadoop.

But the threat is real. Cloudera has partnerships with many analytics vendors, but none of the companies mentioned here are operating up the stack from Cloudera. They’re all addressing the foundational HDFS, Hadoop MapReduce and cluster-management areas where Cloudera presently does business (although IBM and EMC are operating up the stack with analytics software, too). With so many options available — and with Apache Hadoop code open to anyone who wants to use it — every vendor with aspirations of making big money in Hadoop is going to have to work extra hard to convince users they’re adding value worth paying for.

Image courtesy of Flickr user NileGuide.com.

Related content from GigaOM Pro (subscription req’d):

• Putting Big Data to Work: Opportunities for Enterprises
• In Q4, Data Centers, Not the Cloud, Were the Big Story
• Yahoo Still Emceeing a Growing Hadoop Lovefest

"

Cassandra + Hadoop = Brisk by DataStax

Cassandra + Hadoop = Brisk by DataStax: "
I just heard the announcement DataStax, the company offering Cassandra services, made about Brisk a Hadoop and Hive distribution built on top of Cassandra:

Brisk provides integrated Hadoop MapReduce, Hive and job and task tracking capabilities, while providing an HDFS-compatible storage layer powered by Cassandra.

Brisk was announced officially during the MapReduce panel at Structure Big Data event. But it looks like others have already had a chance to hear about Brisk — is there something that I should be doing to hear the “unofficial” announcements?

DataStax has also made available a whitepaper: “Evolving Hadoop into a Low-Latency Data Infrastructure: Unifying Hadoop, Hive and Apache Cassandra for Real-time and Analytics” that you can download from here



Original title and link: Cassandra + Hadoop = Brisk by DataStax (NoSQL databases © myNoSQL)

"

Videos from Hadoop World

Videos from Hadoop World: "
There was one NoSQL conference that I’ve missed and I was really pissed off: Hadoop World. Even if I’ve followed and curated the Twitter feed, resulting in Hadoop World in tweets, the feeling of not being there made me really sad. But now, thanks to Cloudera I’ll be able to watch most of the presentations. Many of them have already been published and the complete list can be found ☞ here.

Based on the twitter activity on that day, I’ve selected below the ones that seemed to have generated most buzz. The list contains names like Facebook, Twitter, eBay, Yahoo!, StumbleUpon, comScore, Mozilla, AOL. And there are quite a few more …

HBase in production at Facebook

Presented by Jonathan Gray (Facebook)



HBase in Production at Facebook, Jonathan Gray, Facebook

The Hadoop Ecosystem at Twitter

Presented by Kevin Weil (Twitter)



The Hadoop Ecosystem at Twitter, Kevin Weil, Twitter

Twitter - Kevin Weil - Hadoop World 2010

Hadoop at eBay

Presented by Anil Madan (eBay)



Hadoop at eBay, Anil Madan, eBay

View more presentations from Cloudera, Inc..

A Fireside Chat: Using Hadoop to Tackle Big Data at comScore

Presented by Martin Hall (Karmasphere) and Will Duckworth (comScore)




A Fireside Chat: Using Hadoop to Tackle Big Data at comScore, Martin Hall, Karmasphere and Will Duckworth, comScore

comScore - Will Duckworth - Hadoop World 2010

ScaleIn Collecting and Querying Log Data in Near Real-time

Presented by Anurag Phadke (Firefox)



ScaleIn Collecting and Querying Log Data in Near Real-time, Anurag Phadke, Firefox

Mozilla - Anurag Phadke - Hadoop World 2010

AOL’s Data Layer

Presented by Ian Holsman (AOL)



AOL’s Data Layer, Ian Holsman, AOL

AOL - Ian Holsman - Hadoop World 2010

Hadoop Based Intelligent Text Information Processing System

Presented by Vaijanath Rao (AOL) and Rohini Uppuluri (AOL)



Intelligent Text Information Processing System, Vaijanath Rao and Rohini Uppuluri, AOL

AOL - Rao & Uppuluri - Hadoop World 2010

Mixing Real-Time Needs and Batch Processing: How StumbleUpon Built an Advertising Platform using HBase and Hadoop

Presented by Jean-Daniel Cryans (StumbleUpon)



Mixing Real-Time Needs and Batch Processing: How StumbleUpon Built an Advertising Platform using HBase and Hadoop, Jean-Daniel C

StumbleUpon - Jean-Daniel Cryans - Hadoop World 2010

Hadoop at Yahoo! Ready for Business

Presented by Arun C. Murthy (Yahoo!)



Hadoop at Yahoo! Ready for Business, Arun C. Murthy, Yahoo!

Yahoo! - Arun Murthy - Hadoop World 2010

Apache ZooKeeper at Yahoo!

Presented by Mahadev Konar (Yahoo!)



Apache ZooKeeper at Yahoo!, Mahadev Konar, Yahoo

Yahoo! - Mahadev Konar - Hadoop World 2010

And having in mind names like Bank of America, Orbitz, CME, Infochimps, sematext, I bet you can find ☞ many more. So, I guess now we have videos for at least a few days.


Thanks Cloudera!


Original title and link: Videos from Hadoop World (NoSQL databases © myNoSQL)

"