Saturday, 5 March 2011

XCP 1.0 released

XCP 1.0 released: "

After 16 months of development, Xen.org is proud to present the first full version of the Xen Cloud Platform. We wanted to thank the project team, who made this happen.


A full feature list as well as the install image and source packages can be found on the download page.


The following new features and improvements have been added since the XCP 0.5 release of XCP last summer:


  • Includes Xen hypervisor version 3.4.2
  • Includes Linux 2.6.32 privileged domain
  • VM Protection and Recovery: configure scheduled snapshots and (optional) archive of virtual machines via snapshot or export
  • Local host storage caching of VM images to reduce load on shared storage
  • Boot from SAN with multipathing support: boots Xen Hypervisor hosts with HBAs from a SAN, with multipathing support.
  • Improved Linux guest support: Ubuntu templates, Fedora 13/Red Hat Enterprise Linux (RHEL) 6 templates, RHEL / CentOS / Oracle Enterprise Linux versions 5.0 to 5.5 support with a generic “RHEL 5″ template
  • Enhanced guest OS support for Windows 7 SP1, Windows Server 2008 R2 SP1, Windows Server 2003, and Suse Linux Enterprise Server (SLES) 11 SP1
  • Improved MPP RDAC multipathing including path health reporting and alerting through XAPI
  • Snapshot improvements: improved reclamation of space after VM snapshots are deleted, even if the VM is running
  • Support for blktap2 disk backend driver rather than blktap1
  • Support for Citrix XenCenter 5.6 FP1 Windows-based GUI management tool (see here)
  • Support for Openstack Bexar release

XCP is significant for Xen.org for a number of reasons: it allows the Xen.org community to develop interesting new functionality against a mature, stable and scalable virtualization stack. If you do want to get involved, check out the project’s wish list and get in touch with the XCP team via the mailing list.


Although XCP can be used as a stand-alone solution to build private clouds or as an enterprise server virtualization solution, there are significant opportunities to extend, innovate and build on top of XCP. Check out the list of open source projects and commercial solutions which already do this.


XCP integrates seamlessly with the Openstack Bexar release: this means that the Xen Hypervisor and XCP are part of an end-to-end open source software stack covering all components from the bare metal to cloud orchestration software. Over the last year, you have seen the Xen community more closely working with downstream Linux and Qemu. The same is happening with upstream projects such as Openstack and OpenNebula.


Unlike the Xen Hypervisor project, XCP delivers an installable binary. This represents a step-change in usability and enables the Xen developer community to more directly engage with its users.


You can find more information on XCP on the XCP home page and on the Wiki. And thank you again, to everybody who made this release happen!

"

Tuesday, 1 March 2011

Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming

Paper: An Experimental Investigation of the Akamai Adaptive Video Streaming: "

Video is hot on the Internet and people are really interested in knowing how to make it work. Dan Rayburn has a post pointing to a fascinating paper: An Experimental Investigation of the Akamai Adaptive Video Streaming, which talks in some detail about the protocols big players like YouTube, Skype and Akamai use to serve video over on an inherently video unfriendly medium like the Internet. For Akamai they found:


  1. Each video is encoded in five versions at different bit rates and stored in separate files.
  2. The client sends commands to the server with an average inter departure time of about 2 s, i.e. the control algorithm is executed on average each 2 seconds. 
  3. Akamai uses only the video level to adapt the video source to the available bandwidth, whereas the frame rate of the video is kept constant.
  4. When a sudden drop in the available bandwidth occurs, short interruptions of the video playback can occur due to the a large actuation delay.
  5. For a sudden increase of the available bandwidth, the transient time to match the new bandwidth is roughly 150 seconds.

Abstract:



"

Friday, 11 February 2011

InfoQ: NoSQL Shake-Up. Membase and CouchOne merge into Couchbase

InfoQ: NoSQL Shake-Up. Membase and CouchOne merge into Couchbase: "The companies describe the merger as a real synergy or optimal fit. Membase will enhance CouchDB performance by providing an efficient, scalable and distributed caching layer and speeding up view server operations. CouchDB replaces the Membase persistence store (SQLLite) and enhances Membase with querying, indexing, map-reduce and more database capabilities. It also benefits from the more mature operations and tools support that Membase offers. The combination of both allows the new products to serve many more different customer needs than before and scale up to millions of users or down to a single mobile device."

Two Books Alike in Dignity - ACM Queue

Two Books Alike in Dignity - ACM Queue

Monday, 7 February 2011

Does Google do "research"?

Does Google do "research"?: "I've been asked a lot by folks recently about whether the work I'm doing now at Google is 'research' and whether one can really have a 'research career' at Google. This has also led to a lot of interesting discussions about what the role of research is in an industrial setting. TL;DR -- yes, Google does research, but not like any other company I know.

Here's my personal take on what 'research' means at Google. (Don't take this as any official statement -- and I'm sure not everyone at Google would agree with this!)

They don't give us lab coats like this, though I wish they did.
The conventional model for industrial research is to set up a lab populated entirely by PhDs, whose job is mostly to write papers, and (in the best case) inform the five-to-ten year roadmap for the company. Usually the 'research lab' is a separate entity from the product side of the company, and may even be physically remote.

Under this model, it can be difficult to get anything you build into production. I have a lot of friends and colleagues at places like Microsoft Research and Intel Labs, and they readily admit that 'technology transfer' is not always easy. Of course, it's not their job to build real systems -- it's primarily to build prototypes, write papers about those prototypes, and then move on to the next big thing. Sometimes, rarely, a research project will mature to the point where it gets picked up by the product side of the company, but this is the exception rather than the norm. It's like throwing ping pong balls at a mountain -- it takes a long time to make a dent.

But these labs aren't supposed to be writing code that gets picked up directly by products -- it's about informing the long-term strategic direction. And a lot of great things can come out of that model. I'm not knocking it -- I spent a year at Intel Research Berkeley before joining Harvard, so I have some experience with this style of industrial research.

From what I can tell, Google takes a very different approach to research. We don't have a separate 'research lab.' Instead, research is distributed throughout the many engineering efforts within the company. Most of the PhDs at Google (myself included) have the job title 'software engineer,' and there's generally no special distinction between the kinds of work done by people with PhDs versus those without. Rather than forking advanced projects off as a separate “research” activity, much of the research happens in the course of building Google’s core systems. Because of the sheer scales at which Google operates, a lot of what we do involves research even if we don't always call it that.

There is also an entity called Google Research, which is not a separate physical lab, but rather a distributed set of teams working in areas such as machine learning, information retrieval, natural language processing, algorithms, and so forth. It's my understanding that even Google Research builds and deploys real systems, like Google’s automatic language translation and voice recognition platforms.

(Update 23-Jan-2011: Someone pointed out that Google also has a 'Quantitative Analyst' job role. These folks work closely with teams in engineering and research to analyze massive data sets, build models, and so forth -- a lot of this work results in research publications as well.)

I like the Google model a lot, since it keeps research and engineering tightly integrated, and keeps us honest. But there are some tradeoffs. Some of the most common questions I've fielded lately include:

Can you publish papers at Google? Sure. Google publishes hundreds of research papers a year. (Some more details here.)You can even sit on program committees, give talks, attend conferences, all that. But this is not your main job, so it's important to make sure that the research outreach isn't interfering with your ability to do get 'real' work done. It's also true that Google teams are sometimes too busy to spend much time pushing out papers, even when the work is eminently publishable.

Can you do long-term crazy beard-scratching pie-in-the-sky research at Google? Maybe. Google does some crazy stuff, like developing self-driving cars. If you wanted to come to Google and start an effort to, say, reinvent the Internet, you'd have to work pretty hard to convince people that it could be done and makes sense for the company. Fortunately, in my area of systems and networking, I don't need to look that far out to find really juicy problems to work on.

Do you have to -- gulp -- maintain your code? And write unit tests? And documentation? And fix bugs? Oh yes. All of that and more. And I love it. Nothing gets me going more than adding a feature or fixing a bug in my code when I know that it will affect millions of people. Yes, there is overhead involved in building real production systems. But knowing that the systems I build will have immediate impact is a huge motivator. So, it's a tradeoff.

But doesn't it bother you that you don't have a fancy title like 'distinguished scientist' and get your own office? I thought it would bug me, but I'm actually quite proud to be a lowly software engineer. I love the open desk seating, and I'm way more productive in that setting. It's also been quite humbling to work side by side with these hotshot developers who are only a couple of years out of college and know way more than I do about programming.

I will be frank that Google doesn't always do the best job reaching out to folks with PhDs or coming from an academic background. When I interviewed (both in 2002 and in 2010), I didn't get a good sense of what I could contribute at Google. The software engineering interview can be fairly brutal: I was asked questions about things I haven't seen since I was a sophomore in college. And a lot of people you talk to will tell you (incorrectly) that 'Google doesn't do research.' Since I've been at Google for a few months, I have a much better picture and one of my goals is to get the company to do a better job at this. I'll try to use this blog to give some of the insider view as well.

Obligatory disclaimer: This is my personal blog. The views expressed here are mine alone and not those of my employer.
"