Top Posts of 2013: Big Data Beyond MapReduce: Google's Big Data Papers | Architects Zone

Source: http://architects.dzone.com/articles/big-data-beyond-mapreduce

Mainstream Big Data is all about MapReduce, but when looking at real-time data, limitations of that approach are starting to show. In this post, I'll review Google's most important Big Data publications and discuss where they are (as far as they've disclosed).

MapReduce, Google File System and Bigtable: the mother of all big data algorithms

Chronologically the first paper is on the Google File System  from 2003, which is a distributed file system. Basically, files are split into chunks which are stored in a redundant fashion on a cluster of commodity machines (Every article about Google has to include the term "commodity machines"!)
Next up is the MapReduce  paper from 2004. MapReduce has become synonymous with Big Data. Legend has it that Google used it to compute their search indices. I imagine it worked like this: They have all the crawled web pages sitting on their cluster and every day or so they ran MapReduce to recompute everything.
Next up is the Bigtable  paper from 2006 which has become the inspiration for countless NoSQL databases like Cassandra, HBase, and others. About half of the architecture of Cassandra is modeled after BigTable, including the data model, SSTables, and write-through-logs (the other half being Amazon's Dynamo database for the peer-to-peer clustering model).

Percolator: Handling individual updates

Google didn't stop with MapReduce. In fact, with the exponential growth of the Internet, it became impractical to recompute the whole search index from scratch. Instead, Google developed a more incremental system, which still allowed for distributed computing.
Now here is where it's getting interesting, in particular compared to what common messages from mainstream Big Data are. For example, they have reintroduced transactions, something NoSQL still tells you that you don't need or cannot have if you want to have scalability.
In the Percolator  paper from 2010, they describe how the Google is keeping its web search index up to date. Percolator is built on existing technologies like Bigtable, but adds transactions and locks on rows and tables, as well as notifications for change in the tables. These notifications are then used to trigger the different stages in a computation. This way, the individual updates can "percolate" through the database.
This approach is reminiscent of stream processing frameworks (SPFs) like Twitter's Storm , or Yahoo's S4 , but with an underlying data base. SPFs usually use message passing and no shared data. This makes it easier to reason about what is happening, but also has the problem that there is no way to access the result of the computation unless you manually store it somewhere in the end.

Pregel: Scalable graph computing

Eventually, Google also had to start mining graph data like the social graph in an online social network, so they developed Pregel , published in 2010.
The underlying computational model is much more complex than in MapReduce: Basically, you have worker threads for each node which are run in parallel iteratively. In each so-called superstep, the worker threads can read messages in the node's inbox, send messages to other nodes, set and read values associated with nodes or edges, or vote to halt. Computations are run till all nodes have voted to halt. In addition, there are also Aggregators and Combiners which compute global statistics.
The paper shows how to implement a number of algorithms like Google's PageRank, shortest path, or bipartite matching. My personal feeling is that Pregel requires even more rethinking on the side of the implementor than MapReduce or SPFs.

Dremel: Online visualizations

Finally, in another paper from 2010, Google describes Dremel , which is an interactive database with an SQL-like language for structured data. So instead of tables with fixed fields like in an SQL database, each row is something like a JSON object (of course, Google uses it's own protocol buffer  format). Queries are pushed down to servers and then aggregated on their way back up and use some clever data format for maximum performance.

Big Data beyond MapReduce

Google didn't stop with MapReduce, but they developed other approaches for applications where MapReduce wasn't a good fit, and I think this is an important message for the whole Big Data landscape. You cannot solve everything with MapReduce. You can make it faster by getting rid of the disks and moving all the data to in-memory, but there are tasks whose inherent structure makes it hard for MapReduce to scale.
Open source projects have picked up on the more recent ideas and papers by Google. For example, ApacheDrill  is reimplementing the Dremel framework, while projects like Apache Giraph  and Stanford's GPS  are inspired by Pregel.
There are still other approaches as well. I'm personally a big fan of stream mining (not to be confused with stream processing) which aims to process event streams with bounded computational resources by resorting to approximation algorithms. Noel Welsh has some interesting slide's  on the topic.

Sent from Evernote

Using RequireJS with Angular - Inline Block's Blog

Using RequireJS With Angular

Source: http://blog.inlineblock.com/blog/2013/06/06/using-requirejs-with-angular/

Since attending Fluent Conf 2013 and watching the many AngularJS talks and seeing the power of its constructs, I wanted to get some experience with it.

Most of the patterns for structuring the code for single page webapps, use some sort dependency management for all the JavaScript instead of using global controllers or other similar bad things. Many of the AngularJS examples seem to follow these bad-ish patterns. Using angular.module('name' , []), helps this problem (why don't they show more angular.module() usage in their tutorials?), but you can still end up with a bunch of dependency loading issues (at least without hardcoding your load order in your header). I even spent time talking to a few engineers with plenty experience with Angular and they all seemed to be okay with just using something like Ruby's asset pipeline to include your files (into a global scope) and to make sure everything ends up in one file in the end via their build process. I don't really like that, but if you are fine with that, I'd suggest you do what you are most comfortable with.

Why RequireJS?

I love using RequireJS. You can async load your dependencies and basically remove all globals from your app. You can use r.js to compile all your JavaScript into a single file and minify that easily, so that your app loads quickly.

So how does this work with Angular? You'd think it would be easy when making single page web apps. You need your 'module' aka your app. You add the routing to your app but to have your routing, you need the controllers and to have your controllers you need the module they belong to. If you do not structure your code and what you load in with Require.js in the right order, you end up with circular dependencies.

Example

So below for my directory structure. My module/app is called "mainApp".

My base public directory:

directory listing

123456789101112131415

index.html- javascripts - controllers/ - directives/ - factories/ - modules/ - routes/ - templates/ - vendors/ require.js jquery.js main.js require.js- stylesheets/ ...

Here is my boot file, aka my main.js.

javascripts/main.js

12345678910111213141516171819

require.config({ baseUrl: '/javascripts', paths: { 'jQuery': '//ajax.googleapis.com/ajax/libs/jquery/1.10.1/jquery.min', 'angular': '//ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular', 'angular-resource': '//ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular-resource' }, shim: { 'angular' : {'exports' : 'angular'}, 'angular-resource': { deps:['angular']}, 'jQuery': {'exports' : 'jQuery'} }});require(['jQuery', 'angular', 'routes/mainRoutes'] , function ($, angular, mainRoutes) { $(function () { // using jQuery because it will run this even if DOM load already happened angular.bootstrap(document , ['mainApp']); });});

You'll notice how I am not loading my mainApp in. Basically we are bringing in the last thing that needs to configured for your app to load, to prevent circular dependencies. Since the Routes need the mainApp controllers and the controllers need the mainApp module, we just have them directly include the mainApp.js.

Also we are configuring require.js to bring in angular and angular-resource (angular-resource so we can do model factories).

Here is my super simple mainApp.js

javascripts/modules/mainApp.js

123	define(['angular' , 'angular-resource'] , function (angular) { return angular.module('mainApp' , ['ngResource']);});

And here is my mainRoutes file:

javascripts/routes/mainRoutes.js

123456

define(['modules/mainApp' , 'controllers/listCtrl'] , function (mainApp) { return mainApp.config(['$routeProvider' , function ($routeProvider) { $routeProvider.when('/' , {controller: 'listCtrl' , templateUrl: '/templates/List.html'}); }]);});

You will notice I require the listCtrl, but actually use its reference. Including it adds it to my mainApp module so it can be used.

Here is my super simple controller:

javascripts/controllers/listCtrl.js

12345

define(['modules/mainApp' , 'factories/Item'] , function (mainApp) { mainApp.controller('listCtrl' , ['$scope' , 'Item' , function ($scope , Item) { $scope.items = Item.query(); });});

So you'll notice, I have to include that mainApp again, so I can add the controller to it. I also have a dependency on Item, which in this case is a factory.The reason I include that, is so that it gets added to the app, so the dependency injection works. Again, I don't actually reference it, I just let dependency injection do its thing.

Lets take a look at this factory really quick.

javascripts/factories/Item.js

12345

define(['modules/mainApp'] , function (mainApp) { mainApp.factory('Item' , ['$resource' , function ($resource) { return $resource('/item/:id' , {id: '@id'}); }]);});

Pretty simple, but again, we have to pull in that mainApp module to add the factory to it.

So finally lets look at our index.html, most if it is simple stuff, but the key part is the ng-view portion, which tells angular where to place the view. Even if you don't use document in your bootstrap and opt to use a specific element, you still need this ng-view.

index.html

123456789101112

<!DOCTYPE html><html><head> <title>Angular and Require</title> <script src="/javascripts/require.js" data-main="javascripts/main"></script></head><body> <div class="page-content"> <ng:view></ng:view> </div></body></html>

Posted by Inline Block Jun 6th, 2013 amd, angular, angularjs, coding, javascript, requirejs

Like

Share

3 people like this. Be the first of your friends.

« TDD JavaScript with Require.js and Teabag on Rails I don't really like Twitter Ads »

Sent from Evernote

PayPal Switches from Java to JavaScript

by Abel Avram on Nov 29, 2013 | Discuss
PayPal has decided to use JavaScript from browser all the way to the back-end server for web applications, giving up legacy code written in JSP/Java.
Jeff Harrell, Director of Engineering at PayPal, has explained in a couple of blog posts (Set My UI Free Part 1: Dust JavaScript Templating, Open Source and More , Node.js at PayPal ) why they decided and some conclusions resulting from switching their web application development from Java/JSP to a complete JavaScript/Node.js stack.
According to Harrell, PayPal's websites had accumulated a good deal of technical debt, and they wanted a "technology stack free of this which would enable greater product agility and innovation." Initially, there was a significant divide between front-end engineers working in web technologies and back-end ones coding in Java. When a UX person wanted to sketch up some pages, they had to ask Java programmers to do some back-end wiring to make it work. This did not fit with their Lean UX development model:

At the time, our UI applications were based on Java and JSP using a proprietary solution that was rigid, tightly coupled and hard to move fast in. Our teams didn't find it complimentary to our Lean UX development model and couldn't move fast in it so they would build their prototypes in a scripting language, test them with users, and then later port the code over to our production stack.

They wanted a "templating [solution that] must be decoupled from the underlying server technology and allow us to evolve our UIs independent of the application language" and that would work with multiple environments. They decided to go with Dust.js  – a templating framework backed up by LinkedIn – , plus Twitter's Bootstrap  and Bower , a package manager for the web. Additional pieces added later were LESS , RequireJS , Backbone.js , Grunt , and Mocha .
Some of PayPal's pages have been redesigned but they still had some of the legacy stack:

… we have legacy C++/XSL and Java/JSP stacks, and we didn't want to leave these UIs behind as we continued to move forward. JavaScript templates are ideal for this. On the C++ stack, we built a library that used V8 to perform Dust renders natively – this was amazingly fast! On the Java side, we integrated Dust using a Spring ViewResolver coupled with Rhino to render the views.

At that time, they also started using Node.js for prototyping new pages, concluding that it was "extremely proficient" and decided to try it in production. For that they also built Kraken.js , a "convention layer" placed on top of Express  which is a Node.js-based web framework. (PayPal has recently open sourced Kraken.js.) The first application to be done in Node.js was the account overview page, which is one of the most accessed PayPal pages, according to Harrell. But because they were afraid the app might not scale well, they decided to create an equivalent Java application to fall back to in case the Node.js one won't work. Following are some conclusions regarding the development effort required for both apps:

	Java/Spring	JavaScript/Node.js
Set-up time	0	2 months
Development	~5 months	~3 months
Engineers	5	2
Lines of code	unspecified	66% of unspecified

The JavaScript team needed 2 months for the initial setup of the infrastructure, but they created with fewer people an application with the same functionality in less time. Running the test suite on production hardware, they concluded that the Node.js app was performing better than the Java one, serving:

Double the requests per second vs. the Java application. This is even more interesting because our initial performance results were using a single core for the node.js application compared to five cores in Java. We expect to increase this divide further.

and having

35% decrease in the average response time for the same page. This resulted in the pages being served 200ms faster— something users will definitely notice.

As a result, PayPal began using the Node.js application in beta in production, and have decided that "all of our consumer facing web applications going forward will be built on Node.js," while some of the existing ones are being ported to Node.js.
One of the benefits of using JavaScript from browser to server is, according to Harrell, the elimination of a divide between front and back-end development by having one team "which allows us to understand and react to our users' needs at any level in the technology stack."

Tell us what you think

Sent from Evernote

List of 20+ Sentiment Analysis APIs | Mashape Blog

List of 20+ Sentiment Analysis APIs Posted 7 months ago

  Powered by

Just a few days back we posted a List of 40+ Machine Learning APIs .
The APIs below are a Sentiment Analysis subset group from that Machine Learning API list.  Sentiment Analysis refers to "the application of natural language processing, computational linguistics, and text analytics to identify and extract subjective information in source materials."
We hope you'll it find useful!

1. S entiment Analysis for Social Media  - The multilingual sentiment analysis API (with exceptional accuracy, 83.4% as opposed to industry standard of 65.4%, and available in Mandarin) from Chatterbox classifies social media texts as positive or negative, with a free daily allowance to get you started. The system uses advanced statistical models (machine learning & NLP) trained on social data, meaning the detection can handle slang, common misspellings, emoticons, hashtags, etc.
2. Text-Processing - Sentiment analysis, stemming and lemmatization, part-of-speech tagging and chunking, phrase extraction and named entity recognition.
3. ML Analyzer - Text Classification, Article Summarization, Sentiment Analysis, Stock symbol extraction, Person Names Extractor, Language Detection, Locations Extractor, Adult content Analyzer.
4. Anger Detection for Social Media  - This unique API will revolutionise your service levels, protect your brand and monitor both sales and promotional campaigns. Designed specifically for social media this API automatically measures the anger levels within social messages so you can quickly highlight action points. Combined with Chatterbox Sentiment Analysis, Anger Detection is designed to protect your brand and service interaction with an online audience.
5. TweetSentiments - Returns the sentiment of Tweets. Two online APIs call the Twitter API to analyze Tweets from a given Twitter user or Tweets returned by a Twitter search query. The offline API analyzes texts of Tweets you've already got, one Tweet at a time.
6. Repustate Sentiment and Social Media Analytics - Repustate's sentiment analysis and social media analytics API allows you to extract key words and phrases and determine social media sentiment in one of many languages. These languages include English, Arabic, German, French and Spanish. Monitor social media as well using our API and retrieve your data all with simple API calls.
7. Chinese Sentiment Analysis for Social Media - 此API适用于中文社交媒体的情感分析（例如新浪微博），能针对每一条消息进行情感分类：正面或负面。该系统基于社交媒体，能够充分利用俚语，特殊词语等新新网络用语。请注意：该免费版本提供每天500条消息分类 - 超过此上限，将会被额外收费。
8. Viralheat Sentiment - Viralheat sentiment is free API and allows users to submit short chunks of text for sentiment scoring.
9. Text Processing - The WebKnox text processing API lets you process (natural) language texts. You can detect the text's language, the quality of the writing, find entity mentions, tag part-of-speech, extract dates, extract locations, or determine the sentiment of the text.
10. Skyttle  - Skyttle API is designed to turn any text into constituent terms (meaningful expressions), entities (names of people, place and things), and sentiment terms. Languages supported are English, Spanish, French, German, Chinese, Swedish, Greek, Czech, Italian and Russian.
11. Fluxifi NLP - Cloud based Natural Language Processing API. Includes Sentiment and Language Detection.
12. Sentiment Analysis Spanish - Sentiment analysis for Spanish language of any given tweet.
13. AlchemyAPI - AlchemyAPI provides advanced cloud-based and on-premise text analysis infrastructure that eliminates the expense and difficulty of integrating natural language processing systems into your application, service, or data processing pipeline.
14. nlpTools - Text processing framework to analyse Natural Language. It is especially focused on text classification and sentiment analysis of online news media (general-purpose, multiple topics).
15. Chinese Analytics - Soshio allows companies to quickly expand their understanding of the Chinese market. Its Chinese Analytics API provides Chinese text analytics and sentiment analysis capabilities for businesses to create their own social monitoring dashboard.
16. Truthy - Write scripts to work with our data, statistics, and images using the API. Download tweet volume over time, network layout, and statistics about memes and users, such as predicted political partisanship, sentiment score, language, and activity.
17. Speech2Topics - Yactraq Speech2Topics is a cloud service that converts audiovisual content into topic metadata via speech recognition & natural language processing. Customers use Yactraq metadata to target ads, build UX features like content search/discovery and mine Youtube videos for brand sentiment. In the past such services have been expensive and only used by large video publishers. The unique thing about Yactraq is we deliver our service at a price any product developer can afford.
18. Bitext Sentiment Analysis - The purpose of this service is to extract opinions from text. An opinion represents the subject an author is writing about and a sentiment score that classifies how positively or negatively the author feels towards that subject. Deep Linguistic Analysis is used to identify the subject the author is discussing.  
19. Textalytics Sentiment Analysis - Multilingual sentiment analysis of texts from different sources (blogs, social networks,…). Besides polarity at sentence and global level, Textalytics Sentiment Analysis 1.1 uses advanced natural language processing techniques to also detect the polarity associated to both entities and concepts in the text. Sentiment Analysis also gives the user the possibility of detecting the polarity of user-defined entities and concepts, making the service a flexible tool applicable to any kind of scenario.
20. Sentiment - This tool works by examining individual words and short sequences of words (n-grams) and comparing them with a probability model. The probability model is built on a prelabeled test set of IMDb movie reviews. It can also detect negations in phrases, i.e, the phrase "not bad" will be classified as positive despite having two individual words with a negative sentiment.
21. Starget sentiment analysis - his is a short text (a twitt or a single sentence) sentiment classification API. It has two types of analysis: one for finding more (but less accurate) sentiment snippets and another one for finding more accurate sentiment (but missing some difficult cases). 
22. Textalytics Media Analysis - Textalytics Media Analysis API analyzes mentions, topics, opinions and facts in all types of media. This API provides services for: - Sentiment analysis - Extracts positive and negative opinions according to the context.
23. Nevahold - Nevahold is a customer service application that leverages the social influence of its community to help users get their voice heard by companies. This API gives you real-time information of company's - Response Time on Facebook and Twitter - Average Response Rate - Customer Service Score - Sentiments - Geo Locations of interactions - Trending keywords
24. Free Natural Language Processing Service - 100% free service including sentiment analysis, content extraction, and language detection. Enjoy!

You should also check out our other useful API lists for machine learning , natural language processing , summarizing text , SMS APIs , and face recognition APIs .

Sent from Evernote