Unlimited-Data

How to Write High-Performance C# Code

2012-02-13T13:43:00.002+01:00

http://dotnet.sys-con.com/node/46342 Writing code that runs quickly is sometimes at odds with writing code quickly. C.A.R. Hoare, computer science luminary and discoverer of the QuickSort algorithm, famously proclaimed, "Premature optimization is the root of all evil." The extreme programming design principle of "You Aren't Gonna Need It" (YAGNI) argues against implementing any features,

High Scalability - High Scalability - Are Cloud Based Memory Architectures the Next Big Thing?

2012-01-30T22:33:00.001+01:00

High Scalability - High Scalability - Are Cloud Based Memory Architectures the Next Big Thing?: We are on the edge of two potent technological changes: Clouds and Memory Based Architectures. This evolution will rip open a chasm where new players can enter and prosper. Google is the master of disk. You can't beat them at a game they perfected. Disk based databases like SimpleDB and BigTable are

In-Memory Database Systems Questions and Answers

2012-01-11T22:30:00.001+01:00

In-Memory Database Systems Questions and Answers: In-Memory Database Systems - Questions and AnswersIn-memory database systems (IMDS) are a growing sub-set of a database management system (DBMS) software. In-memory databases emerged in response to new application goals, system requirements, and operating environments. Below, we answer common IMDS questions. What is an in-memory database system

Notes on Scalability | Brent Ozar PLF

2012-01-11T20:54:00.001+01:00

Notes on Scalability | Brent Ozar PLF: We all hope that we’re going to succeed beyond our wildest expectations. Startups long for multi-billion dollar IPOs or scaling to hundreds, or even thousands, of servers. Every hosting provider is touting how their new cloud offering will help us scale up to unheard of heights. I’ve built things up and torn them down a few times over my career Build it to

How Facebook built its Timeline feature

2012-01-08T01:19:00.001+01:00

How Facebook built its Timeline feature: Facebook’s Timeline feature is beautiful, although some revile it. But love it or hate it, Timeline is the engineering equivalent of building a racing bike customized for a specific track, only without testing either until race day. At least that’s how Ryan Mack, an infrastructure engineer with Facebook, makes it seem on a blog posted Thursday detailing

Paper: Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS

2011-11-25T12:41:00.001+01:00

onPaper: Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS: Teams from Princeton and CMU are working together to solve one of the most difficult problems in the repertoire: scalable geo-distributed data stores. Major companies like Google and Facebook have been working on multiple datacenter database functionality for some time, but there's still a general

High Scalability - High Scalability - Using Gossip Protocols for Failure Detection, Monitoring, Messaging and Other Good Things

2011-11-15T11:16:00.001+01:00

High Scalability - High Scalability - Using Gossip Protocols for Failure Detection, Monitoring, Messaging and Other Good Things: Using Gossip Protocols For Failure Detection, Monitoring, Messaging And Other Good ThingsMONDAY, NOVEMBER 14, 2011 AT 9:15AMWhen building a system on top of a set of wildly uncooperative and unruly computers you have knowledge problems: knowing when other nodes are dead

High Scalability - High Scalability - VoltDB Decapitates Six SQL Urban Myths and Delivers Internet Scale OLTP in the Process

2011-10-25T11:53:00.001+02:00

High Scalability - High Scalability - VoltDB Decapitates Six SQL Urban Myths and Delivers Internet Scale OLTP in the Process: VoltDB Decapitates Six SQL Urban Myths And Delivers Internet Scale OLTP In The ProcessMONDAY, JUNE 28, 2010 AT 7:36AMWhat do you get when you take a SQL database and start a new implementation from scratch, taking advantage of the latest research and modern hardware?

Presentation: HBase @ Facebook

2011-09-30T11:47:00.000+02:00

Presentation: HBase @ Facebook: Kannan Muthukkaruppan overviews HBase, explaining what Facebook Messages is and why they chose HBase to implement it, their contribution to HBase, and what they plan to use it for in the future. By Kannan Muthukkaruppan

Presentation: Building Scalable Systems: an Asynchronous Approach

2011-09-22T15:35:00.001+02:00

Presentation: Building Scalable Systems: an Asynchronous Approach: Theo Schlossnagle expresses his opinion on Big Data, NoSQL, cloud, system architecture and design, then he discusses the benefit of using asynchronous queues for building scalable systems. By Theo Schlossnagle

Programming != Computer Science

2011-09-15T16:55:00.001+02:00

Programming != Computer Science: I recently read this very interesting article on ways to "level up" as a software developer. Reading this article brought home something that has been nagging me for a while since joining Google: that there is a huge skill and cultural gap between "developers" and "Computer Scientists." Jason's advice to leveling-up in the aforementioned article is very practical:

OpenStack turns 1. What’s next? — Cloud Computing News

2011-09-06T15:13:00.001+02:00

OpenStack turns 1. What’s next? — Cloud Computing News: OpenStack, the open-source, cloud-computing software project founded by Rackspace and NASA, celebrates its first birthday tomorrow. It has been a busy year for the project, which appears to have grown much faster than even its founders expected it would. A year in, OpenStack is still picking up steam and looks not only like an open source

High Scalability - High Scalability - The Three Ages of Google - Batch, Warehouse, Instant

2011-09-06T11:50:00.002+02:00

High Scalability - High Scalability - The Three Ages of Google - Batch, Warehouse, Instant: The world has changed. And some things that should not have been forgotten, were lost. I found these words from the Lord of the Rings echoing in my head as I listened to a fascinating presentation by Luiz André Barroso, Distinguished Engineer at Google, concerning Google's legendary past, golden present,

LevelDB - Fast and Lightweight Key/Value Database From the Authors of MapReduce and BigTable

2011-08-11T00:09:00.001+02:00

LevelDB - Fast and Lightweight Key/Value Database From the Authors of MapReduce and BigTable: " LevelDB is an exciting new entrant into the pantheon of embedded databases, notable both for its pedigree, being authored by the makers of the now mythical Google MapReduce and BigTable products, and for its emphasis on efficient disk based random access using log-structured-merge (LSM) trees. The

Hadoop's tremendous inefficiency on graph data management (and how to avoid it)

2011-08-05T16:00:00.001+02:00

Hadoop's tremendous inefficiency on graph data management (and how to avoid it): "Hadoop is great. It seems clear that it will serve as the basis of the vast majority of analytical data management within five years. Already today it is extremely popular for unstructured and polystructured data analysis and processing, since it is hard to find other options that are superior from a price/

Is Stonebraker right? Why SQL isn’t the choice du jour for many apps

2011-08-03T14:08:00.001+02:00

Is Stonebraker right? Why SQL isn’t the choice du jour for many apps: " Two weeks ago, I wrote a post that sparked a pretty overwhelming response. The gist of the post, derived from an interview with database pioneer Michael Stonebraker, was that legacy SQL databases, including MySQL, are relics and no longer relevant with regard to today’s web applications. Stonebraker cited Facebook’s renowned

Facebook trapped in MySQL ‘fate worse than death’

2011-08-03T13:34:00.000+02:00

Facebook trapped in MySQL ‘fate worse than death’: " According to database pioneer Michael Stonebraker, Facebook is operating a huge, complex MySQL implementation equivalent to “a fate worse than death,” and the only way out is “bite the bullet and rewrite everything.” Not that it’s necessarily Facebook’s fault, though. Stonebraker says the social network’s predicament is all too common among

SIGMOD 2011 in Athens

2011-06-26T23:25:00.004+02:00

SIGMOD 2011 in Athens: " Earlier this week, I was in Athens Greece attending annual conference of the ACM Machinery Special Interest Group on Management of Data. SIGMOD is one of the top two database events held each year attracting academic researchers and leading practitioners from industry. I kicked off the conference with the Plenary keynote. In this talk I started with a short

What Larry Page really needs to do to return Google to its startup roots | Slacy’s Blog

2011-06-26T23:17:00.000+02:00

What Larry Page really needs to do to return Google to its startup roots | Slacy’s Blog What Larry Page really needs to do to return Google to its startup roots I worked at Google from 2005-2010, and saw the company go through many changes, and a huge increase in staff. Most importantly, I saw the company go from a place where engineers were seen as violent disruptors and innovators, to a place

Google BigTable, MapReduce, MegaStore vs. Hadoop, MongoDB

2011-06-26T23:16:00.000+02:00

Google BigTable, MapReduce, MegaStore vs. Hadoop, MongoDB: "Google BigTable, MapReduce, MegaStore vs. Hadoop, MongoDB: Dhanji R. Prasanna leaving Google: Here is something you’ve may have heard but never quite believed before: Google’s vaunted scalable software infrastructure is obsolete. Don’t get me wrong, their hardware and datacenters are the best in the world, and as far as I know, nobody

High Scalability - High Scalability - 35+ Use Cases for Choosing Your Next NoSQL Database

2011-06-23T00:35:00.001+02:00

High Scalability - High Scalability - 35+ Use Cases for Choosing Your Next NoSQL Database We've asked What The Heck Are You Actually Using NoSQL For?. We've asked 101 Questions To Ask When Considering A NoSQL Database. We've even had a webinar What Should I Do? Choosing SQL, NoSQL or Both for Scalable Web Applications.Now we get to the point of considering use cases and which systems might be

LexisNexis open-sources its Hadoop killer

2011-06-18T14:03:00.001+02:00

LexisNexis open-sources its Hadoop killer: "LexisNexis is releasing a set of open-source, data-processing tools that it says outperforms Hadoop and even handles workloads Hadoop presently can’t. The technology (and new business line) is called HPCC Systems, and was created 10 years ago within the LexisNexis Risk Solutions division that analyzes huge amounts of data for its customers in

InfoQ: Netflix’s Cloud Data Architecture

2011-06-14T14:24:00.002+02:00

InfoQ: Netflix’s Cloud Data Architecture Siddharth Anand overviews Netflix’s business model, then he explains why they chose Amazon AWS, and how they moved their data into the cloud using a NoSQL solution.

Apple iCloud: Syncing and Distributed Storage Over Streaming and Centralized Storage

2011-06-07T22:29:00.001+02:00

Apple iCloud: Syncing and Distributed Storage Over Streaming and Centralized Storage: " There has been a lot of speculation over how Apple's iCloud would work. With the Apple Worldwide Developers Conference keynotes having just completed, we finally learned the truth. We can handle it. They made some interesting and cost effective architecture choices that preserved the value of their devices and

Awesome List of Advanced Distributed Systems Papers

2011-06-03T11:22:00.001+02:00

Awesome List of Advanced Distributed Systems Papers: " As part of Dr. Indranil Gupta's CS 525 Spring 2011 Advanced Distributed Systems class, he has collected an incredible list of resources on distributed systems. His research group is also doing some interesting work. The various topics include: Before there Were Clouds, Cloud Computing, P2P Systems, Basic Distributed Computing Concepts, Sensor

Stuff to Watch from Surge 2010

2011-05-30T15:47:00.001+02:00

Stuff to Watch from Surge 2010: " Surge is a conference put on by OmniTI targeting practical Scalability matters. OmniTI specializes in helping people solve their scalability problems, as is only natural, as it was founded by Theo Schlossnagle, author of the canonical Scalable Internet Architectures. Now that Surge 2011 is on the horizon, they've generously made available nearly all the videos

Updated AWS Security White Paper; New Risk and Compliance White Paper

2011-05-24T09:36:00.001+02:00

Updated AWS Security White Paper; New Risk and Compliance White Paper: "We have updated the AWS Security White Paper and we've created a new Risk and Compliance White Paper. Both are available now. The AWS Security White Paper describes our physical and operational security principles and practices. It includes a description of the shared responsibility model, a summary of our control

How can academics do research on cloud computing?

2011-05-11T00:50:00.001+02:00

How can academics do research on cloud computing?: "This week I'm in Napa for HotOS 2011 -- the premier workshop on operating systems. HotOS is in its 24th year -- it started as the Workshop on Workstation Operating Systems in 1987. More on HotOS in a forthcoming blog post, but for now I wanted to comment on a very lively argument discussion that took place during the panel session yesterday. The

EMC Makes a Big Bet on Hadoop

2011-05-10T20:13:00.001+02:00

EMC Makes a Big Bet on Hadoop: "EMC is throwing its weight behind Hadoop. Today, at EMC World, the storage giant announced a slew of Hadoop-centric products, including a specialized appliance for Hadoop-based big data analytics and two separate Hadoop distributions. EMC’s entry is most definitely going to shake up the Hadoop and database markets. EMC is now the largest company actively pushing

High Scalability - High Scalability - Comet - An Example of the New Key-Code Databases

2011-05-10T15:12:00.001+02:00

High Scalability - High Scalability - Comet - An Example of the New Key-Code Databases Comet is an active distributed key-value store built at the University of Washington. The paper describing Comet is Comet: An active distributed key-value store, there are also slides, and a MP3 of a presentation given at OSDI '10. Here's a succinct overview of Comet: Today's cloud storage services, such as

Presentation:More Best Practices for Large-Scale Websites: Lessons from eBay

2011-05-09T21:23:00.002+02:00

Presentation:More Best Practices for Large-Scale Websites: Lessons from eBay: "Randy Shoup shares 10 lessons learned from eBay: Partition Everything, Asynchrony Everywhere, Automate Everything, Everything Fails, Embrace Inconsistency, Expect (R)evolution, Dependencies Matter, Respect Authority, Never Enough Data, Custom Infrastructure. By Randy Shoup" Summary Randy Shoup shares 10 lessons

Presentation:More Best Practices for Large-Scale Websites: Lessons from eBay

2011-05-09T21:23:00.000+02:00

Stack Overflow Makes Slow Pages 100x Faster by Simple SQL Tuning

2011-05-02T22:40:00.002+02:00

Stack Overflow Makes Slow Pages 100x Faster by Simple SQL Tuning: " Source: highscalability.com The most common complaint against NoSQL is that if you know how to write good SQL queries then SQL works fine. If SQL is slow you can always tune it and make it faster. A great example of this incremental improvement process was written up by StackExchange's Sam Saffron, in A day in the life of a

The Updated Big List of Articles on the Amazon Outage

2011-05-02T22:36:00.001+02:00

The Updated Big List of Articles on the Amazon Outage: "Source: Highscalability.com Since The Big List Of Articles On The Amazon Outage was published we've a had few updates that people might not have seen. Amazon of course released their Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region. Netlix shared their Lessons Learned from the AWS Outage as did Heroku (How

InfoQ: Evolution of Code Design at Facebook

2011-04-29T15:52:00.001+02:00

InfoQ: Evolution of Code Design at Facebook

ACM books 04/2011

2011-04-29T15:41:00.002+02:00

Dear Viet-Trung Tran, This is your monthly ACM Technical Interest Service email, providing access to highlights of current ACM activities and information that you can tailor to match your interests. This program is a free ACM Student Membership benefit. This issue features books from ACM's collection of 500 online titles, powered by Books24/7®. The book offerings have been divided into six "

The Rise of Hadoop: How many Hadoop-related solutions exist?

2011-04-29T00:51:00.001+02:00

The Rise of Hadoop: How many Hadoop-related solutions exist?: "The Rise of Hadoop: How many Hadoop-related solutions exist?: The CMSWire commented list of Hadoop-related solutions: Apache Hadoop Appistry CloudIQ Storage Hadoop Edition: a HDFS replacement improving on the single NameNode ( here). Shipping. IBM Distribution of Apache Hadoop: Apache Hadoop, a 32-bit Linux version of the IBM SDK

Scalable SQL - ACM Queue

2011-04-27T01:20:00.001+02:00

Scalable SQL - ACM Queue

Paper: NoSQL Databases - NoSQL Introduction and Overview

2011-04-13T22:35:00.001+02:00

Paper: NoSQL Databases - NoSQL Introduction and Overview: "Christof Strauch, from Stuttgart Media University, has written an incredible 120+ page paper titled NoSQL Databases as an introduction and overview to NoSQL databases . The paper was written between 2010-06 and 2011-02, so it may be a bit out of date, but if you are looking to take in the NoSQL world in one big gulp, this is your chance.

MapReduce: A major step backwards | The Database Column

2011-04-12T11:51:00.001+02:00

MapReduce: A major step backwards | The Database Column MapReduce: A major step backwardson Jan 17 in Database architecture, Database history, Database innovation posted by DeWitt [Note: Although the system attributes this post to a single author, it was written by David J. DeWitt and Michael Stonebraker]On January 8, a Database Column reader asked for our views on new distributed database

How Facebook Changed Technology in One Day

2011-04-08T10:11:00.001+02:00

How Facebook Changed Technology in One Day: " The biggest deal about Facebook’s open compute project isn’t the project, it’s the wave of innovation this can bring forward at the systems level — which will affect everyone from the chipmakers to the giant systems vendors and data center operators. At its event Thursday, Facebook unveiled the Open Compute Project, which essentially open sources the

Introduction to Architecting Systems for Scale

2011-04-05T18:38:00.001+02:00

Introduction to Architecting Systems for Scale: " Few computer science or software development programs attempt to teach the building blocks of scalable systems. Instead, system architecture is usually picked up on the job by working through the pain of a growing product or by working with engineers who have already learned through that suffering process. In this post I'll attempt to document

Presentation:Parallel Programming Patterns: Data Parallelism

2011-04-03T19:10:00.001+02:00

Presentation:Parallel Programming Patterns: Data Parallelism: "Ralph Johnson presents several data parallelism patterns, including related Java, C# and C++ libraries from Intel and Microsoft, comparing it with other forms of parallelism such as actor programming. By Ralph Johnson"

NoSQL & Cloud at Netflix

2011-04-01T10:28:00.001+02:00

NoSQL & Cloud at Netflix: "NoSQL & Cloud at Netflix: Today Netflix can be seen as a leader in what can be achieved by combining cloud computing and polyglot persistence. Not only that, but Netflix has chosen to share their experience with everyone else so we can all learn from their experience. Netflix’s experience of migrating from an on-premise architecture using relational databases has been

Music to my Ears - Introducing Amazon Cloud Drive and Amazon Cloud Player

2011-03-29T13:05:00.001+02:00

Music to my Ears - Introducing Amazon Cloud Drive and Amazon Cloud Player: " Today Amazon.com announced new solutions to help customers manage their digital music collections. Amazon Cloud Drive and Amazon Cloud Player enable customers to securely and reliably store music in the cloud and play it on any Android phone, tablet, Mac or PC, wherever they are. As a big music fan with well over

As Big Data Takes Off, the Hadoop Wars Begin

2011-03-29T11:44:00.001+02:00

As Big Data Takes Off, the Hadoop Wars Begin: " It turns out “big data” isn’t just a buzzword, but a legitimate concern for companies across the board. Their interest in the tools to take advantage of the opportunity for analysis of all this data has sparked a land grab among established vendors and startups alike. The action is centered around Hadoop, the flagship technology for storing and

Cassandra + Hadoop = Brisk by DataStax

2011-03-24T10:10:00.001+01:00

Cassandra + Hadoop = Brisk by DataStax: " I just heard the announcement DataStax, the company offering Cassandra services, made about Brisk a Hadoop and Hive distribution built on top of Cassandra: Brisk provides integrated Hadoop MapReduce, Hive and job and task tracking capabilities, while providing an HDFS-compatible storage layer powered by Cassandra. Brisk was announced officially

InfoQ: Application Security With Apache Shiro

2011-03-20T22:00:00.001+01:00

InfoQ: Application Security With Apache Shiro

InfoQ: QCon Keynote: Innovation at Google

2011-03-20T21:31:00.001+01:00

InfoQ: QCon Keynote: Innovation at Google

6 Lessons from Dropbox - One Million Files Saved Every 15 minutes

2011-03-15T17:47:00.001+01:00

6 Lessons from Dropbox - One Million Files Saved Every 15 minutes: " Dropbox saves one million files every 15 minutes, more tweets than even Twitterers tweet. That mind blowing statistic was revealed by Rian Hunter, a Dropbox Engineer, in his presentation How Dropbox Did It and How Python Helped at PyCon 2011. The first part of the presentation is some Dropbox lore, origin stories and other

InfoQ: Riak Core: Dynamo Building Blocks

2011-03-11T16:59:00.001+01:00

InfoQ: Riak Core: Dynamo Building Blocks

Great work at FAST ’11

2011-03-06T23:17:00.001+01:00

Great work at FAST ’11: "After a quick scan of the paper titles I wasn’t impressed. But after seeing presentations and posters I am. Here’s some I found interesting. I’ll be posting longer pieces on some of these. A Study of Practical Deduplication Full paper *Best Paper Winner* Tradeoffs in Scalable Data Routing for Deduplication Clusters Full paper Exploiting Half-Wits: Smarter Storage for

XCP 1.0 released

2011-03-05T16:18:00.001+01:00

XCP 1.0 released: "After 16 months of development, Xen.org is proud to present the first full version of the Xen Cloud Platform. We wanted to thank the project team, who made this happen. A full feature list as well as the install image and source packages can be found on the download page. The following new features and improvements have been added since the XCP 0.5 release of XCP last summer:

2011-03-05T16:17:00.000+01:00

Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming

2011-03-01T23:34:00.001+01:00

Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming: "Video is hot on the Internet and people are really interested in knowing how to make it work. Dan Rayburn has a post pointing to a fascinating paper: An Experimental Investigation of the Akamai Adaptive Video Streaming, which talks in some detail about the protocols big players like YouTube, Skype and Akamai use to

TechPack - Cloud Computing

2011-02-28T16:00:00.000+01:00

TechPack - Cloud Computing

InfoQ: NoSQL Shake-Up. Membase and CouchOne merge into Couchbase

2011-02-11T11:38:00.001+01:00

InfoQ: NoSQL Shake-Up. Membase and CouchOne merge into Couchbase: "The companies describe the merger as a real synergy or optimal fit. Membase will enhance CouchDB performance by providing an efficient, scalable and distributed caching layer and speeding up view server operations. CouchDB replaces the Membase persistence store (SQLLite) and enhances Membase with querying, indexing, map-reduce and

Two Books Alike in Dignity - ACM Queue

2011-02-11T11:34:00.001+01:00

Two Books Alike in Dignity - ACM Queue

Does Google do "research"?

2011-02-07T22:32:00.001+01:00

Does Google do "research"?: "I've been asked a lot by folks recently about whether the work I'm doing now at Google is 'research' and whether one can really have a 'research career' at Google. This has also led to a lot of interesting discussions about what the role of research is in an industrial setting. TL;DR -- yes, Google does research, but not like any other company I know. Here's my

Why Netflix Picked Amazon SimpleDB, Hadoop/HBase, and Cassandra

2011-02-01T19:49:00.001+01:00

Why Netflix Picked Amazon SimpleDB, Hadoop/HBase, and Cassandra: "Why Netflix Picked Amazon SimpleDB, Hadoop/HBase, and Cassandra: Yury Izrailevsky[1]: The reason why we use multiple NoSQL solutions is because each one is best suited for a specific set of use cases. For example, HBase is naturally integrated with the Hadoop platform, whereas Cassandra is best for cross-regional deployments and

InfoQ: Asynchronous, Event-Driven Web Servers for the JVM: Deft and Loft

2011-01-26T20:51:00.001+01:00

InfoQ: Asynchronous, Event-Driven Web Servers for the JVM: Deft and Loft Asynchronous, event-driven architectures have been gaining a lot of attention lately, mostly with respect toJavaScript and Node.js. Deft and Loft are two solutions that bring "asynchronous purity" to the JVM.RelatedVendorContentDomain-Specific Languages for Functional TestingFuture of Java Track - EE, Spring, JVMs+ @

Google Megastore: The Data Engine Behind GAE

2011-01-18T15:49:00.001+01:00

Google Megastore: The Data Engine Behind GAE: " Megastore is the data engine supporting the Google Application Engine. It’s a scalable structured data store providing full ACID semantics within partitions but lower consistency guarantees across partitions. I wrote up some notes on it back in 2008 Under the Covers of the App Engine Datastore and posted Phil Bernstein’s excellent notes

Riak's Bitcask - A Log-Structured Hash Table for Fast Key/Value Data

2011-01-18T15:37:00.001+01:00

Riak's Bitcask - A Log-Structured Hash Table for Fast Key/Value Data: " How would you implement a key-value storage system if you were starting from scratch? The approach Basho settled on with Bitcask, their new backend for Riak, is an interesting combination of using RAM to store a hash map of file pointers to values and a log-structured file system for efficient writes. In this excellent

Google Megastore - 3 Billion Writes and 20 Billion Read Transactions Daily

2011-01-18T15:34:00.001+01:00

Google Megastore - 3 Billion Writes and 20 Billion Read Transactions Daily: " A giant step into the fully distributed future has been taken by the Google App Engine team with the release of their High Replication Datastore. The HRD is targeted at mission critical applications that require data replicated to at least three datacenters, full ACID semantics for entity groups, and lower consistency

A Plea to Software Vendors from Sysadmins - 10 Do's and Don'ts - ACM Queue

2011-01-18T15:30:00.001+01:00

A Plea to Software Vendors from Sysadmins - 10 Do's and Don'ts - ACM Queue What can software vendors do to make the lives of sysadmins a little easier?THOMAS A. LIMONCELLI, GOOGLE A friend of mine is a grease monkey: the kind of auto enthusiast who rebuilds engines for fun on a Saturday night. He explained to me that certain brands of automobiles were designed in ways to make the mechanic's job

Armitage – Cyber Attack Management & GUI For Metasploit

2010-12-14T21:36:00.001+01:00

Armitage – Cyber Attack Management & GUI For Metasploit: "Armitage is a graphical cyber attack management tool for Metasploit that visualizes your targets, recommends exploits, and exposes the advanced capabilities of the framework. Armitage aims to make Metasploit usable for security practitioners who understand hacking but don’t use Metasploit every day. If you want to learn Metasploit and grow

Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3

2010-12-14T01:01:00.001+01:00

Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3: " Today, Amazon S3 announced a new breakthrough in supporting customers with large files by increasing the maximum supported object size from 5 gigabytes to 5 terabytes. This allows customers to store and reference a large file as a single object instead of smaller 'chunks'. When combined with the Amazon S3 Multipart Upload release,

Amazon Route 53 DNS Service

2010-12-14T00:19:00.001+01:00

Amazon Route 53 DNS Service: " Even working in Amazon Web Services, I’m finding the frequency of new product announcements and updates a bit dizzying. It’s amazing how fast the cloud is taking shape and the feature set is filling out. Utility computing has really been on fire over the last 9 months. I’ve never seen an entire new industry created and come fully to life this fast. Fun times.

Expanding the Cloud with DNS - Introducing Amazon Route 53

2010-12-09T17:09:00.001+01:00

Expanding the Cloud with DNS - Introducing Amazon Route 53: "I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. DNS is one of the fundamental building blocks of internet applications and was high on the wish list of our customers for some time already. Route 53 has the business properties that you have come to

Kafka : A high-throughput distributed messaging system.

2010-12-05T14:55:00.001+01:00

Kafka : A high-throughput distributed messaging system.: "Found an interesting new open source project which I hadn’t heard about before. Kafka is a messaging system used by linkedin to serve as the foundation of their activity stream processing.Kafka is a distributed publish-subscribe messaging system. It is designed to support the followingPersistent messaging with O(1) disk structures that

Almost half of cloud revenues from storage!

2010-12-05T14:44:00.001+01:00

Almost half of cloud revenues from storage!: "A new report from the 451 Group says that the cloud computing marketplace will reach $16.7bn in revenue by 2013. Even more interesting, however, the Group reports the cloud-based storage will play a starring role in cloud growth, accounting for nearly 40% of the core cloud pie in 2010. “We view storage as the most fertile sector, and predict that

Pragmatic Programming Techniques: Scalable System Design Patterns

2010-12-05T14:41:00.000+01:00

Pragmatic Programming Techniques: Scalable System Design Patterns Looking back after 2.5 years since my previous post on scalable system design techniques, I've observed an emergence of a set of commonly used design patterns. Here is my attempt to capture and share them. Load Balancer In this model, there is a dispatcher that determines which worker instance will handle the request based on

The Full Stack, Part I

2010-12-03T17:59:00.001+01:00

The Full Stack, Part I: "One of my most vivid memories from school was the day our chemistry teacher let us in on the Big Secret: every chemical reaction is a joining or separating of links between atoms. Which links form or break is completely governed by the energy involved and the number of electrons each atom has. The principle stuck with me long after I'd forgotten the details. There existed

Design — Sheepdog Project

2010-11-29T23:59:00.002+01:00

Design — Sheepdog Project The architecture of Sheepdog is fully symmetric; there is no central node such as a meta-data server. This design enables following features.Linear scalability in performance and capacityWhen more performance or capacity is needed, Sheepdog can be grown linearly by simply adding new machines to the cluster. No single point of failureEven if a machine fails, the data

20 Things I Learned About Browsers and the Web

2010-11-27T16:54:00.000+01:00

20 Things I Learned About Browsers and the Web

Netflix in the Cloud

2010-11-27T16:42:00.000+01:00

View more presentations from Adrian Cockcroft.

Scalability | Harvard Computer Science Lecture

2010-11-26T16:27:00.001+01:00

Scalability | Harvard Computer Science Lecture: " Watch it on Academic Earth LECTURE DESCRIPTIONProfessor David J. Malan discusses scalability as it pertains to building dynamic websites.COURSE DESCRIPTIONToday's websites are increasingly dynamic. Pages are no longer static HTML files but instead generated by scripts and database calls. User interfaces are more seamless, with technologies like

ZooKeeper Promoted to Apache Top Level Project

2010-11-23T20:54:00.001+01:00

ZooKeeper Promoted to Apache Top Level Project: " ZooKeeper, the centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services, that started under Hadoop umbrella has been promoted to an Apache Top Level Project, according to the ☞ report sent out by Doug Cutting. In case you are wondering what it means, simply put it’s

Another NoSQL Comparison: Evaluation Guide

2010-11-23T20:50:00.001+01:00

Another NoSQL Comparison: Evaluation Guide: "Another NoSQL Comparison: Evaluation Guide: The requirements were clear: Fast data insertion. Extremely fast random reads on large datasets. Consistent read/write speed across the whole data set. Efficient data storage. Scale well. Easy to maintain. Have a network interface. Stable, of course. The list of NoSQL databases to be compared: Tokyo

Videos from Hadoop World

2010-11-16T20:57:00.001+01:00

Videos from Hadoop World: " There was one NoSQL conference that I’ve missed and I was really pissed off: Hadoop World. Even if I’ve followed and curated the Twitter feed, resulting in Hadoop World in tweets, the feeling of not being there made me really sad. But now, thanks to Cloudera I’ll be able to watch most of the presentations. Many of them have already been published and the complete list

The Underlying Technology of Messages

2010-11-16T20:21:00.001+01:00

The Underlying Technology of Messages: " We're launching a new version of Messages today that combines chat, SMS, email, and Messages into a real-time conversation. The product team spent the last year building out a robust, scalable infrastructure. As we launch the product, we wanted to share some details about the technology. The current Messages infrastructure handles over 350 million users

Strategy: Biggest Performance Impact is to Reduce the Number of HTTP Requests

2010-11-16T01:53:00.001+01:00

Strategy: Biggest Performance Impact is to Reduce the Number of HTTP Requests: " Low Cost, High Performance, Strong Security: Pick Any Three by Chris Palmer has a funny and informative presentation where the main message is: reduce the size and frequency of network communications, which will make your pages load faster, which will improve performance enough that you can use HTTPS all the time,

Facebook: The Underlying Technology of Messages Using HBase

2010-11-16T01:44:00.001+01:00

Facebook: The Underlying Technology of Messages Using HBase: "Facebook: The Underlying Technology of Messages Using HBase: Cassandra, MySQL, and HBase were evaluated for the Facebook new messaging system. HBase was finally picked and is now behind the new product announced today: HBase comes with very good scalability and performance for this workload and a simpler consistency model than

LISA '10 Technical Sessions

2010-11-13T23:27:00.002+01:00

LISA '10 Technical Sessions Data Structures from the Future: Bloom Filters, Distributed Hash Tables, and More! Thomas A. Limoncelli,Google, Inc. View the SlidesGreetings, earthlings of the year 2010! I've traveled back in time to share with you some of the technologies that system administrators operate in the future. Chances are you know what a cache is and how to tune it. In the future,

Quick Reference: Hadoop Tools Ecosystem

2010-11-11T17:19:00.001+01:00

Quick Reference: Hadoop Tools Ecosystem: " Just a quick reference of the continuously growing Hadoop tools ecosystem. Hadoop The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing. hadoop.apache.org HDFS A distributed file system that provides high throughput access to application data. hadoop.apache.org/hdfs/ MapReduce A software framework

MongoDB Replica Sets

2010-11-11T17:15:00.001+01:00

MongoDB Replica Sets: " Even if MongoDB replica sets ☞ official documentation is quite good, that doesn’t mean more coverage of the subject is not going to be useful. So, this is kind of everything you need to read about MongoDB replica sets. Let’s start with Kristina Chodorow’s series on MongoDB replica sets: ☞ Replica sets part1: Master-Slave is so 2009 This post shows how to do the “Hello

Presentation:Machine Learning: A Love Story

2010-11-09T22:49:00.003+01:00

Presentation:Machine Learning: A Love Story: "Hilary Mason presents the history of machine learning covering some of the most significant developments taking place over the last two decades, especially the fundamental math and algorithmic tools employed. She also exemplifies how machine learning is used by bit.ly to discover various statistical information about users. By Hilary Mason"

Introducing S4

2010-11-09T19:09:00.001+01:00

Stream of WordsMotivationData streams abound in the world of Big Data: Twitter, search queries, stock quotes, website analytics, sensor data to name a few. Yet, popular approaches for data processing at this scale are based on MapReduce: a batch-oriented framework; in other cases, there are proprietary stream processing systems, or ad-hoc solutions for particular problems.We understand that the

Running the Show: Configuration Management with Chef: RailsConf 2009 - O'Reilly Conferences, May 04 - 07, 2009, Las Vegas, NV!

2010-10-28T17:46:00.001+02:00

RUNNING THE SHOW: CONFIGURATION MANAGEMENT WITH CHEFEdd Dumbill (O'Reilly Media, Inc. )8:30am Monday, 05/04/2009Tutorial Location: Pavilion 2 - 3Presentations: Running the Show_ Configuration Management with Chef Presentation [PDF], Running the Show_ Configuration Management with Chef Presentation 1 [ZIP] Few completed Rails apps are architecturally simple. As soon as you grow, you find

From Zero to the Cloud with Todd Deshane and Patrick F. Wilbur � Linux, Todd, Open, LISA, Patrick, Cloud � USENIX Update

2010-10-27T23:55:00.001+02:00

From Zero to the Cloud with Todd Deshane and Patrick F. Wilbur � Linux, Todd, Open, LISA, Patrick, Cloud � USENIX UpdateTodd Deshane: Sure. I’m a recent graduate from Clarkson University, where I got a Ph.D in Engineering Science; I’ve been a member of xen.orgcommunity for quite sometime, basically ever since it came out. Patrick Wilbur: I’m a graduate student, Ph.D in Computer Science at

Google search index splits with MapReduce • The Register

2010-10-26T23:33:00.001+02:00

Google search index splits with MapReduce Welds BigTable to file system 'Colossus'By Cade Metz in San Francisco • Get more from this authorPosted in Servers, 9th September 2010 21:52 GMTFree whitepaper – Trying to keep smartphones off your network?Exclusive Google Caffeine — the remodeled search infrastructure rolled out across Google's worldwide data center network earlier this year — is not

Dawn of a New Day « Ray Ozzie

2010-10-26T23:07:00.001+02:00

Dawn of a New Day « Ray Ozzie Dawn of a New DayTo: Executive Staff and direct reports Date: October 28, 2010 From: Ray Ozzie Subject: Dawn of a New DayFive years ago, having only recently arrived at the company, I wrote The Internet Services Disruption in order to kick off a major change management process across the company. In the opening section of that memo,

Writing Parsers in Ruby using Treetop

2010-10-23T21:40:00.001+02:00

Writing Parsers in Ruby using Treetop: " Treetop is one of the most underrated, yet powerful, Ruby libraries out there. If you want to write a parser, it kicks ass. The only problem is unless you're into reading up about and playing with parsers, it's not always obvious how to get going with them, or Treetop in particular. Luckily Aaron Gough, Toronto-based Ruby developer, comes to our rescue

The CAP Theorem... Again

2010-10-23T16:31:00.001+02:00

The CAP Theorem... Again: " Today looks to be (again) the day of the CAP theorem[1][2], so let’s do a quick summary: We had Coda Hale’s ☞ You can’t sacrifice partition tolerance: Of the CAP theorem’s Consistency, Availability, and Partition Tolerance, Partition Tolerance is mandatory in distributed systems. You cannot not choose it. Instead of CAP, you should think about your availability in

What is Network-based Application Virtualization and Why Do You Need It?

2010-10-23T16:28:00.001+02:00

What is Network-based Application Virtualization and Why Do You Need It?: " With all the attention being paid these days to VDI (virtual desktop infrastructure) and application virtualization and server virtualization and virtualization it’s easy to forget about network-based application virtualization. But it’s the one virtualization technique you shouldn’t forget because it is a

RESTful Cassandra

2010-10-21T13:14:00.001+02:00

RESTful Cassandra: "RESTful Cassandra: Gary Dusbabek: A lot of people, when first learning about Cassandra, wonder why there isn’t any easier (say, RESTful) way to perform operations. I did. It didn’t take someone very long to point out that it mainly has to do with performance. Cassandra spends a significant amount of resources marshaling data and Thrift currently does that very

Objectively speaking: the future of objects

2010-10-19T10:57:00.001+02:00

Objectively speaking: the future of objects: " One infrastructure to rule them all discussed the emerging enterprise need for a single, scalable file storage infrastructure. But what infrastructure? Some background to this is last year’s Cloud Quadrant and this year’s Why private clouds are part of the future. Block and file For decades direct-attached block-based storage was the only

Everyday GIT With 20 Commands Or So

2010-10-14T10:20:00.001+02:00

Everyday GIT With 20 Commands Or So Everyday GIT With 20 Commands Or So[Individual Developer (Standalone)] commands are essential for anybody who makes a commit, even for somebody who works alone.If you work with other people, you will need commands listed in the [Individual Developer (Participant)] section as well.People who play the [Integrator] role need to learn some more commands in addition

Hadoop: The Definitive Guide, Second Edition

2010-10-11T20:04:00.001+02:00

Hadoop: The Definitive Guide, Second Edition: " The second edition of my book “Hadoop: The Definitive Guide”, published by O’Reilly, is now available. The first edition was launched at the Hadoop Summit in June 2009, and has gone on to sell well. Less than a year later I was asked to write the second edition. The Hadoop ecosystem has been growing fast (and continues to), and the bulk of the extra

MapReduce: A major step backwards | The Database Column

2010-10-08T00:04:00.000+02:00