Unlimited-Data. moved to lab.itbee.vn

Thursday, 23 June 2011

High Scalability - High Scalability - 35+ Use Cases for Choosing Your Next NoSQL Database

We've asked What The Heck Are You Actually Using NoSQL For?. We've asked 101 Questions To Ask When Considering A NoSQL Database. We've even had a webinar What Should I Do? Choosing SQL, NoSQL or Both for Scalable Web Applications.

Now we get to the point of considering use cases and which systems might be appropriate for those use cases.

What Are Your Options?

First, let's cover what are the various data models. These have been adapted from Emil Eifrem andNoSQL databases.

Document Databases

Lineage: Inspired by Lotus Notes.

Data model: Collections of documents, which contain key-value collections.

Example: CouchDB, MongoDB

Good at: Natural data modeling. Programmer friendly. Rapid development. Web friendly, CRUD.

Graph Databases

Lineage: Euler and graph theory.

Data model: Nodes & relationships, both which can hold key-value pairs

Example: AllegroGraph, InfoGrid, Neo4j

Good at: Rock complicated graph problems. Fast.

Relational Databases

Lineage: E. F. Codd in A Relational Model of Data for Large Shared Data Banks

Data Model: a set of relations

Example: VoltDB, Clustrix, MySQL

Good at: High performing, scalable OLTP. SQL access. Materialized views. Transactions matter. Programmer friendly transactions.

Object Oriented Databases

Lineage: Graph Database Research

Data Model: Objects

Example: Objectivity, Gemstone

Key-Value Stores

Lineage: Amazon's Dynamo paper and Distributed HashTables.

Data model: A global collection of KV pairs.

Example: Membase, Riak

Good at: Handles size well. Processing a constant stream of small reads and writes. Fast. Programmer friendly.

BigTable Clones

Lineage: Google's BigTable paper.

Data model: Column family, i.e. a tabular model where each row at least in theory can have an individual configuration of columns.

Example: HBase, Hypertable, Cassandra

Goog at: Handles size well. Stream massive write loads. High availability. Multiple-data centers. MapReduce.

Data Structure Servers

Lineage: ?

Example: Redis

Data model: Operations over dictionaries, lists, sets and string values.

Good at: Quirky stuff you never thought of using a database for before.

Grid Databases

Lineage: Data Grid and Tuple Space research.

Data Model: Space Based Architecture

Example: GigaSpaces, Coherence

Good at: High performance and scalable transaction processing.

What Should Your Application Use?

Key point is to rethink how your application could work differently in terms of the different data models and the different products. Right data model for the right problem. Right product for the right problem.

To see what models might help your application take a look at What The Heck Are You Actually Using NoSQL For? In this article I tried to pull together a lot of unconventional use cases of the different qualities and features developers have used in building systems.

Match what you need to do with these use cases. From there you can backtrack to the products you may want to include in your architecture. NoSQL, SQL, it doesn't matter.

Look at Data Model + Product Features + Your Situation. Products have such different feature sets it's almost impossible to recommend by pure data model alone.

Which option is best is determined by your priorities.

If Your Application Needs...

complex transactions because you can't afford to lose data or if you would like a simple transaction programming model then look at a Relational or Grid database.
- Example: an inventory system that might want full ACID. I was very unhappy when I bought a product and they said later they were out of stock. I did not want a compensated transaction. I wanted my item!

to scale then NoSQL or SQL can work. Look for systems that support scale-out, partitioning, live addition and removal of machines, load balancing, automatic sharding and rebalancing, and fault tolerance.

to always be able to write to a database because you need high availability then look at Bigtable Clones which feature eventual consistency.

to handle lots of small continuous reads and writes, that may be volatile, then look at Document or Key-value or databases offering fast in-memory access. Also consider SSD.

to implement social network operations then you first may want a Graph database or second, a database like Riak that supports relationships. An in- memory relational database with simple SQL joins might suffice for small data sets. Redis' set and list operations could work too.

If Your Application Needs...

to operate over a wide variety of access patterns and data types then look at a Document database, they generally are flexible and perform well.

powerful offline reporting with large datasets then look at Hadoop first and second, products that support MapReduce. Supporting MapReduce isn't the same as being good at it.

to span multiple data-centers then look at Bigtable Clones and other products that offer a distributed option that can handle the long latencies and are partition tolerant.

to build CRUD apps then look at a Document database, they make it easy to access complex data without joins.

built-in search then look at Riak.

to operate on data structures like lists, sets, queues, publish-subscribe then look at Redis. Useful for distributed locking, capped logs, and a lot more.

programmer friendliness in the form of programmer friendly data types like JSON, HTTP, REST, Javascript then first look at Document databases and then Key-value Databases.

If Your Application Needs...

transactions combined with materialized views for real-time data feeds then look at VoltDB. Great for data-rollups and time windowing.

enterprise level support and SLAs then look for a product that makes a point of catering to that market. Membase is an example.

to log continuous streams of data that may have no consistency guarantees necessary at all then look at Bigtable Clones because they generally work on distributed file systems that can handle a lot of writes.

to be as simple as possible to operate then look for a hosted or PaaS solution because they will do all the work for you.

to be sold to enterprise customers then consider a Relational Database because they are used to relational technology.

to dynamically build relationships between objects that have dynamic properties then consider a Graph Database because often they will not require a schema and models can be built incrementally through programming.

to support large media then look storage services like S3. NoSQL systems tend not to handle large BLOBS, though MongoDB has a file service.

If Your Application Needs...

to bulk upload lots of data quickly and efficiently then look for a product supports that scenario. Most will not because they don't support bulk operations.

an easier upgrade path then use a fluid schema system like a Document Database or a Key-value Database because it supports optional fields, adding fields, and field deletions without the need to build an entire schema migration framework.

to implement integrity constraints then pick a database that support SQL DDL, implement them in stored procedures, or implement them in application code.

a very deep join depth the use a Graph Database because they support blisteringly fast navigation between entities.

to move behavior close to the data so the data doesn't have to be moved over the network then look at stored procedures of one kind or another. These can be found in Relational, Grid, Document, and even Key-value databases.

If Your Application Needs...

to cache or store BLOB data then look at a Key-value store. Caching can for bits of web pages, or to save complex objects that were expensive to join in a relational database, to reduce latency, and so on.

a proven track record like not corrupting data and just generally working then pick an established product and when you hit scaling (or other issues) use on of the common workarounds (scale-up, tuning, memcached, sharding, denormalization, etc).

fluid data types because your data isn't tabular in nature, or requires a flexible number of columns, or has a complex structure, or varies by user (or whatever), then look at Document, Key-value, and Bigtable Clone databases. Each has a lot of flexibility in their data types.

other business units to run quick relational queries so you don't have to reimplement everything then use a database that supports SQL.

to operate in the cloud and automatically take full advantage of cloud features then we may not be there yet.

If Your Application Needs...

support for secondary indexes so you can look up data by different keys then look at relational databases and Cassandra's new secondary index support.

creates an ever-growing set of data (really BigData) that rarely gets accessed then look at Bigtable Clone which will spread the data over a distributed file system.

to integrate with other services then check if the database provides some sort of write-behind syncing feature so you can capture database changes and feed them into other systems to ensure consistency.

fault tolerance check how durable writes are in the face power failures, partitions, and other failure scenarios.

to push the technological envelope in a direction nobody seems to be going then build it yourself because that's what it takes to be great sometimes.

to work on a mobile platform then look at CouchDB/Mobile couchbase.

Which Is Better?

Moving for a 25% improvement is probably not a reason to go NoSQL.

Benchmark relevancy depends on the use case. Does it match your situation(s)?

Are you a startup that needs to release a product as soon as possible and you are playing around with ideas? Both SQL and NoSQL can make an argument.

Performance may be equal on one box, but what happens when you need N?

Everything has problems, if you look at Amazon forums it's EBS is slow, or my instances won't reply, etc. For GAE it's the datastore is slow or X. Every product which people are using will have problems. Are you OK with the problems of the system you've selected?

Saturday, 18 June 2011

LexisNexis open-sources its Hadoop killer

LexisNexis open-sources its Hadoop killer: "

LexisNexis is releasing a set of open-source, data-processing tools that it says outperforms Hadoop and even handles workloads Hadoop presently can’t. The technology (and new business line) is called HPCC Systems, and was created 10 years ago within the LexisNexis Risk Solutions division that analyzes huge amounts of data for its customers in intelligence, financial services and other high-profile industries. There have been calls for a legitimate alternative to Hadoop, and this certainly looks like one.

According to Armando Escalante, CTO of LexisNexis Risk Solutions, the company decided to release HPCC now because it wanted to get the technology into the community before Hadoop became the de facto option for big data processing. Escalante told me during a phone call that he thinks of Hadoop as “a guy with a machete in front of a jungle — they made a trail,” but that he thinks HPCC is superior.

But in order to compete for mindshare and developers, he said, the company felt it had to open-source the technology. One big thing Hadoop has going for it is its open-source model, Escalante explained, which attracts a lot of developers and a lot of innovation. If his company wanted HPCC to “remain relevant” and keep improving through new use cases and ideas from a new community, the time for release was now and open source had to be the model.

Hadoop, of course, is the Apache Software Foundation project created several years ago by then-Yahoo employee Doug Cutting. It has become a critical tool for web companies — including Yahoo and Facebook — to process their ever-growing volumes of unstructured data, and is fast making its way into organizations of all types and sizes. Hadoop has spawned a number of commercial distributions and products, too, including from Cloudera, EMC and IBM.

How HPCC works

Hadoop relies on two core components to store and process huge amounts of data: the Hadoop Distributed File System and Hadoop MapReduce. However, as Cloudant CEO Mike Miller explained in a post over the weekend, MapReduce is still a relatively complex language for writing parallel-processing workflows. HPCC seeks to remedy this with its Enterprise Control Language.

Escalante says ECL is a declarative, data-centric language that abstracts a lot of the work necessary within MapReduce. For certain tasks that take a thousand lines of code in MapReduce, he said, ECL only requires 99 lines. Furthermore, he explained, ECL doesn’t care how many nodes are in the cluster because the system automatically distributes data across however many nodes are present. Technically, though, HPCC could run on just a single virtual machine. And, says Escalante, HPCC is written in C++ — like the original Google MapReduce on which Hadoop MapReduce is based — which he says makes it inherently faster than the Java-based Hadoop version.

HPCC offers two options for processing and serving data: the Thor Data Refinery Cluster and the Roxy Rapid Data Delivery Cluster. Escalante said Thor — so named for its hammer-like approach to solving the problem — crunches, analyzes and indexes huge amounts of data a la Hadoop. Roxie, on the other hand, is more like a traditional relational database or database warehouse that even can serve transactions to a web front end.

We didn’t go into detail on HPCC’s storage component, but Escalante noted that it does utilize a distributed file system, although it can support a variety of off-node storage architectures and/or local solid-state drives.

He added that in order to ensure LexisNexis wasn’t blinded by “eating its own dogfood,” his team hired a Hadoop expert to kick the tires on HPCC. The consultant was impressed, Escalante said, but did note some shortcomings that the team addressed as it readied the technology for release. It also built a converter for migrating Hadoop applications written in the Pig language to ECL.

Can HPCC Systems actually compete?

The million-dollar question is whether HPCC Systems can actually attract an ecosystem of contributors and users that will help it rise above the status of big data also-ran. Escalante thinks it can, in large part because HPCC already has been proven in production dealing with LexisNexis Risk Solutions’ 35,000 data sources, 5,000 transactions per second and large, paying customers. He added that the company also will provide enterprise licenses and proprietary applications in addition to the open-source code. Plus, it already has potential customers lined up.

It’s often said that competition means validation. Hadoop has moved from a niche set of tools to the core of a potentially huge business that’s growing every day, and even Microsoft has a horse in this race with its Dryad set of big data tools. Hadoop has already proven itself, but the companies and organizations relying on it for the their big data strategies can’t rest on their laurels.

Image courtesy of Flickr user NileGuide.com.

Related content from GigaOM Pro (subscription req’d):

Tuesday, 14 June 2011

InfoQ: Netflix’s Cloud Data Architecture

InfoQ: Netflix’s Cloud Data Architecture
Siddharth Anand overviews Netflix’s business model, then he explains why they chose Amazon AWS, and how they moved their data into the cloud using a NoSQL solution.

Tuesday, 7 June 2011

Apple iCloud: Syncing and Distributed Storage Over Streaming and Centralized Storage

Apple iCloud: Syncing and Distributed Storage Over Streaming and Centralized Storage: "

There has been a lot of speculation over how Apple's iCloud would work. With the Apple Worldwide Developers Conference keynotes having just completed, we finally learned the truth. We can handle it. They made some interesting and cost effective architecture choices that preserved the value of their devices and the basic model of how their existing services work.

A lot of pundits foretold that with all the datacenters Apple was building we would get a streaming music solution. Only one copy of music would be stored and then streamed on demand to everyone. Or they could go the Google brute force method and copy up all a user's music and play it on demand.

Apple did neither. The chose an interesting middle path that's not Google, Amazon, or even MobileMe.

They key idea is you no longer need a PC. Device content is now synced over the air and is managed by the cloud, not your legacy computer. Your data may not even be stored in the cloud, but the whole management, syncing, and control of content is done by the cloud instead of the PC. PCs are now just another device on par with the iPhone and iPad.

What happens to your data depends on the type of data. Apple gives you 5GB of free storage, which doesn't sound like a lot at all. The twist here is purchased music, apps, and books, and photos will not count against free storage because these are stored on your devices. Photos hit the cloud for a maximum of 30 days, which allows your devices 30 days to contact the cloud and download the photos, after that I guess they are lost. All the big data is stored on your devices.

Some smaller content is stored in the cloud. This content is mail, documents, Camera Roll, account information, settings, and other app data. This data is much smaller than photos, videos, and music, so it's a manageable amount of storage per user. It wasn't talked about, but I'd imagine storage could be increased for a price, so any increased storage usage would be funded.

What Apple ended up creating is a syncing model where large content is synced between devices, smaller meta-data type content is stored in the cloud, and shared changeable data like mail is stored in the cloud. The advantages of this approach are:

It's consistent with how iTunes works now. Instead of a PC there's a cloud. No revolution here.
It's cost efficient, which is important because iCloud is free. Storage will increase linearly based on the number of users, not the number of media items, which is a much more manageable curve. Apple is not on the hook for ever increasing amounts of data storage.
Bandwidth usage is bursty during syncing operations, but is otherwise low, other than background notifications, etc. In a streaming system bandwidth usage is continuous and high, which shifts their cost structure to that of being a CDN. The path taken here by Apple skips that whole problem. Most if this will probably be over WiFi instead of 3G so user bandwidth caps can be avoided.
The role for devices is preserved. Devices aren't just a thin client for the cloud. The meat of the application logic is still solidly on the device and not the cloud. Apple can still sell high margin devices and you can get your media on all your devices. And since these devices already have storage, there's no need to duplicate storage in the cloud, which is more economical.
The low cost of Apple's new Scan and Match service is made possible because of the minimal storage and bandwidth costs. They can just keep one copy and push it down to devices for local access. What other vendor has this ability? The devices that people are already used to buying offload this cost and are themselves a profit center.
This will really drive the demand for larger and larger SSD drives. The 1000 image storage limit for photos on your devices will be a big negative.

Low level stuff like how merging happens if conflicting changes are made on different devices was not talked about, but the details will be interesting. There are a still a lot of details that need to be explained about where things are stored, how much can be stored, how they get synced, and how much it will cost. And this is still a very personal service. It's device centric. It's not social, there doesn't appear to be any sharing, which seems a strange oversight.

Apple has taken an interesting middle path in their architecture and there's a lot to learn from. They aren't just storing stuff like Amazon and Google. They aren't creating another streaming service. They are creating a product unique to their ecosystem, an environment users will find difficult to leave.

I should mention that this is my very early impression of how things work taken from the presentation and poking around the iCloud site. It's possible I could have got some of it wrong.

Friday, 3 June 2011

Awesome List of Advanced Distributed Systems Papers

Awesome List of Advanced Distributed Systems Papers: "

As part of Dr. Indranil Gupta's CS 525 Spring 2011 Advanced Distributed Systems class, he has collected an incredible list of resources on distributed systems. His research group is also doing some interesting work.

The various topics include: Before there Were Clouds, Cloud Computing, P2P Systems, Basic Distributed Computing Concepts, Sensor Networks, Overlays and DHTs, Cloud Programming, Cloud Scheduling, Key-Value Stores, Storage, Sensor Net Routing, Geo-Distribution, P2P Apps, In-network processing, Epidemics, Probabilistic Membership Protocols, Distributed Monitoring and Management, Publish-Subscribe/CDNs, Measurement Studies, Old Wine: Stale or Vintage?, In Byzantium, Cloud Pricing, Other Industrial Systems, Structure of Networks, Completing the Circle, Green Clouds, Distributed Debugging, Flash!, The Middle or the End?, Availability-Aware Systems, Design Methodologies, Handling Stress, Sources of unreliability in networks, Handling Stress, Selfish algorithms, Security, Economic Theory, The future of sensor nets?, The End-to-End Approach, Automatic Computing and Inference, Caching, Classical Algorithms, Topology and Naming, Practical theory perspectives, Modular Systems.

That's just the list of topics! For every topic there's the slide deck used to teach the class, a main list of papers and a second list of optional papers. So there's a lot to choose from. Happy reading! If any of the papers really stand out for you, please share.

Monday, 30 May 2011

Stuff to Watch from Surge 2010

Stuff to Watch from Surge 2010: "

Surge is a conference put on by OmniTI targeting practical Scalability matters. OmniTI specializes in helping people solve their scalability problems, as is only natural, as it was founded by Theo Schlossnagle, author of the canonical Scalable Internet Architectures.

Now that Surge 2011 is on the horizon, they've generously made available nearly all the videos from the Surge 2010 conference. A pattern hopefully every conference will follow (only don't wait a year please). We lose a lot of collective wisdom from events not being available online in a timely manner.

In truth, nearly all the talks are on topic and are worth watching, but here are a few that seem especially relevant:

Going 0 to 60: Scaling LinkedIn by Ruslan Belkin, Sr. Director of Engineering, LinkedIn.
- Have you ever wondered what architectures the site like LinkedIn may have used and what insights teams have learned while growing the system from serving just a handful to close to a hundred million of users?
Scaling and Loadbalancing Wikia Across The World by Artur Bergman, VP of Engineering and Operations, Wikia.
- Wikia hosts around a 100 000 wikis using the open source Mediawiki software. In this talk I'll take a tour through the process of taking a legacy source code and turning it into a globally distributed system.
Design for Scale - Patterns, Anti-Patterns, Successes and Failures by Christopher Brown, VP of Engineering, Opscode.
- This isn't your "Gang of Four". Christopher will discuss his experiences building Amazon's EC2 and the Opscode Platform, and the experiences of others designing large-scale online services.
Quantifying Scalability FTW by Neil Gunther, Founder/Principal Consultant, Performance Dynamics.
- Successful scalability requires transforming your data to quantify the cost-benefit of any architectural decisions. In other words: information = measurement + method.
Database Scalability Patterns by Robert Treat Lead Database Architect, OmniTI.
- In Database Scalability Patterns we will attempt to distill all of the information/hype/discussions around scaling databases, and break down the common patterns we've seen dealing with scaling databases.
From disaster to stability: scaling challenges of my.opera.com by Cosimo Streppone, Lead Developer, Opera Software.
- This talk tells the story of these last 3 years. Our successes, our failures, and what remains to be done.
Embracing Concurrency at Scale by Justin Sheehy CTO, Basho Technologies.
- Justin will focus on methods for designing and building robust fundamentally-concurrent distributed systems. We will look at practices that are "common knowledge" but too often forgotten, at old lessons that the software industry at large has somehow missed, and at some general "good practices" and rules that must be thrown away when moving into a distributed and concurrent world.
Scalable Design Patterns by Theo Schlossnagle, Principal/CEO, OmniTI.
- In this talk, we'll take a whirlwind tour though different patterns for scalable architecture design and focus on evaluating if each is the right tool for the job. Topics include load balancing, networking, caching, operations management, code deployment, storage, service decoupling and data management (the RDBMS vs. noSQL argument).
Why Some Architects Almost Never Shard Their Applications by Baron Schwartz, VP of Consulting, Percona.
- "Shard early, shard often" is common advice -- and it's often wrong. In reality, many systems don't have to be sharded.
Scaling myYearbook.com - Lessons Learned From Rapid Growth by Gavin M. Roy, Chief Technology Officer, MyYearbook.
- In this talk Gavin will review the growing pains and methodologies used to handle the consistent growth and demand while affording the rapid development cycles required by the product development team.

Tuesday, 24 May 2011

Updated AWS Security White Paper; New Risk and Compliance White Paper

Updated AWS Security White Paper; New Risk and Compliance White Paper: "

We have updated the AWS Security White Paper and we've created a new Risk and Compliance White Paper. Both are available now.

The AWS Security White Paper describes our physical and operational security principles and practices.

It includes a description of the shared responsibility model, a summary of our control environment, a review of secure design principles, and detailed information about the security and backup considerations related to each part of AWS including the Virtual Private Cloud, EC2, and the Simple Storage Service.

The new AWS Risk and Compliance White Paper covers a number of important topics including (again) the shared responsibility model, additional information about our control environment and how to evaluate it, and detailed information about our certifications and third-party attestations. A section on key compliance issues addresses a number of topics that we are asked about on a regular basis.

The AWS Security team and the AWS Compliance team are complimentary organizations and are responsible for the security infrastructure, practices, and compliance programs described in these white papers. The AWS Security team is headed by our Chief Information Security Officer and is based outside of Washington, DC. Like most parts of AWS, this team is growing and they have a number of open positions:

We also have a number of security-related positions open in Seattle:

-- Jeff;