ŷhat | 10 Books for Data Enthusiasts

Source: http://blog.yhathq.com/posts/ten-data-books.html
August 11, 2013

Over the last few years, I've invested a lot of time exploring various areas of data analysis and software development. Going down the proverbial coding rabbit hole, I've quietly accumulated a lot of books on various subjects.
This is a post about 10 data books that I've gotten a lot of milage out of and that really have legs.

1. Programming Collective Intelligence by Toby Segaran

   Synopsis

   An overview of machine learning and the key algorithms in use today. Each chapter outlines a problem, defines an approach to solving it using a particular algorithm, and then gives you all the sample code you need to solve it.
   

   Why you should read it

   One of my favorite books (non-techincal and technical). I try to re-read it at least once per year. Great explanations of how you can make machine learning useful.
   Everyone has something to learn from PCI. My only criticism--the code is indented with 2 spaces instead of 4. Nitpicky, but annoying. Despite the fact that this is one of the oldest books on the list, it has managed to stay extremely relevant in the ever changing landscape of data analysis tools.
   

2. Machine Learning for Hackers by Drew Conway and John Myles White

   Synopsis

   A series of real world case studies and solutions which use machine learning. This is a very practical approach to machine learning. The visuals are great and there are plenty of code samples to go around. A few of the chapters focusing on text classification/regression are particularly well done.
   

   Why you should read it

   I was on the pre-order list for this one. It was a gruelling 3 months on the waiting list but when it arrived Machine Learning for Hackers didn't disappoint. The code examples are optimized for readability rather than optimization which makes it much easier to follow along in the book (and translate them to other languages if need be). The code examples were also translated into Python , so I've included the Python logo even though it's not actually in the book.
   

3. Super Crunchers by Ian Ayres

   Synopsis

   A collection of stories about data, modeling, and analysis, Super Crunchers tells how data and analysis are used in practice. Some of the examples are a little dated, but the core message stands the test of time.
   

   Why you should read it

   It's a lot higher level than most of the books on this list, and is geared for people who might not actually be doing the analysis or the modeling. Still, Super Crunchers is a great read and if you happen to be an analyst or data scientist, this will give you some insight into how the rest of the world views your work (for better or worsee). The most important takeaway from the book is not neccessarily what algorithms or technologies are being applied, but how they're being applied and how they're changing the way that companies use their data.
   

4. Python for Data Analysis by Wes McKinney

   Synopsis

   A few years ago Wes McKinney took one for the team. He quit his job and wrote pandas, the open source Python package for wrangling data. Naturally Wes is the best person to write the book on pandas. The title may be a little misleading but Python for Data Analysis shows you the ins and outs of using pandas to improve your workflow.
   

   Why you should read it

   pandas is a must have for doing analysis with Python. This book focuses more on munging, wrangling, and formatting data (not modeling which many people incorrectly assume). So if you need brush up on your data wrangling (and you probably do) grab this off the shelf.
   

5. R Cookbook by Paul Teetor

   Synopsis

   Pretty straightforward. A series of recipes for problems frequently encountered when doing analysis. Things like: building a regession model, merging data, imputing values, file i/o, etc.
   

   Why you should read it

   R can be a prickly language. The syntax is a little strange when you first start, everything is in tabular form, and weird stuff just tends to happen in general . This is the perfect book for when you have a question like:
   

   "I just want to loop through a bunch of files and combine them together. I know exactly how I'd to it in Python, but how the heck do I do it in R?"

   I strongly recommend this book if you're learning R, especially if you're coming form another programming language. It'll sit on your desk at work forever and you're guaranteed to pick it up at least a couple times per week.

6. The Signal and the Noise by Nate Silver

   Synopsis

   A great overview of how predictions impact different parts of our lives. The book follows a similar pattern to Super Crunchers, telling stories related to data and prediction, and then tying them all together at the end. A great, quick read for anyone interested in data or analysis.
   

   Why you should read it

   Just because it's on The Internet doesn't mean it's true. Same goes with data. If you stare at a chart for long enough, a trend begins to emerge. The Signal and the Noise does a great job at teaching you when to throw up a warning flag when someone hands you some analysis.
   

7. Visualize This by Nathan Yau

   Synopsis

   This is essentially the first couple years of Nathan Yau's blog, Flowing Data, in book format. There are great code examples to go along with some truely spectacular visuals.
   

   Why you should read it

   You can't show off your work with out some nifty data visuals. This book takes you step by step and shows you how you it's easy to construct great looking charts, maps, and other visuals if you use the right tools.
   

8. ggplot2: Elegant Graphics for Data Analysis by Hadley Wickham

   Synopsis

   The name pretty much sums it up. This book shows you how to use ggplot2 by walking you through some examples and gradually adding complexity.
   

   Why you should read it

   If you're going to use R, you're inevitably going to be using ggplot2. ggplot2 is one the most popular R packages and probably the standard for making great looking visualizations. Who better to teach you how to use ggplot2 than the package's creator, Hadley Wickham. The book provides some core examples for making basic plots, and then exapnds on each of these by detailing some of the more in depth and advanced features of ggplot2 which makes it great for both beginners and advanced users.
   

9. The NLTK Books by Jacob Perkins, Steven Bird, Ewan Klein, and Edward Loper

   Synopsis

   The Natural Language Toolkit (NLTK) is an excellent Python library for processing text and language. It has excellent APIs that can preproces, classify, and help analyze your text. The Cookbook and the freely available online book serve as the instruction manuals for using NLTK.
   

   Why you should read it

   Text analytics is really fun. Some of the examples in the NLTK books are really just magical (the text classification chapter is particularly cool ). Some of the code examples use a lot of the Python syntactic sugar which can make it a little difficult to read for someone who is new to Python, but the breadth of examples more than makes up for it. Top it all off with a really amazing library and it makes for a great read.
   

10. Think Stats by Allen B. Downey

    Synopsis

    This book provides a gentle overview to statistics and a nice tutorial on using Python as well. It's sort of a crash course in statistics for those of us who chose to major in something less mathy in school.
    

    Why you should read it

    It's short, sweet, and to the point. Think Stats serves as the introduction to statistics course that many people missed out on in school. If you need to brush up on CDFs, PDFs, Normal Variates, or the Central Limit Theorem, then this is the book you're looking for. Also not a bad way to learn Python while picking up some stats skills.
    

Other Books

A few others that didn't quite make the list but we still love:
Let us know if there are any others you think we missed!

Sent from Evernote

List of 20+ Sentiment Analysis APIs | Mashape Blog

List of 20+ Sentiment Analysis APIs Posted 7 months ago

  Powered by

Just a few days back we posted a List of 40+ Machine Learning APIs .
The APIs below are a Sentiment Analysis subset group from that Machine Learning API list.  Sentiment Analysis refers to "the application of natural language processing, computational linguistics, and text analytics to identify and extract subjective information in source materials."
We hope you'll it find useful!

1. S entiment Analysis for Social Media  - The multilingual sentiment analysis API (with exceptional accuracy, 83.4% as opposed to industry standard of 65.4%, and available in Mandarin) from Chatterbox classifies social media texts as positive or negative, with a free daily allowance to get you started. The system uses advanced statistical models (machine learning & NLP) trained on social data, meaning the detection can handle slang, common misspellings, emoticons, hashtags, etc.
2. Text-Processing - Sentiment analysis, stemming and lemmatization, part-of-speech tagging and chunking, phrase extraction and named entity recognition.
3. ML Analyzer - Text Classification, Article Summarization, Sentiment Analysis, Stock symbol extraction, Person Names Extractor, Language Detection, Locations Extractor, Adult content Analyzer.
4. Anger Detection for Social Media  - This unique API will revolutionise your service levels, protect your brand and monitor both sales and promotional campaigns. Designed specifically for social media this API automatically measures the anger levels within social messages so you can quickly highlight action points. Combined with Chatterbox Sentiment Analysis, Anger Detection is designed to protect your brand and service interaction with an online audience.
5. TweetSentiments - Returns the sentiment of Tweets. Two online APIs call the Twitter API to analyze Tweets from a given Twitter user or Tweets returned by a Twitter search query. The offline API analyzes texts of Tweets you've already got, one Tweet at a time.
6. Repustate Sentiment and Social Media Analytics - Repustate's sentiment analysis and social media analytics API allows you to extract key words and phrases and determine social media sentiment in one of many languages. These languages include English, Arabic, German, French and Spanish. Monitor social media as well using our API and retrieve your data all with simple API calls.
7. Chinese Sentiment Analysis for Social Media - 此API适用于中文社交媒体的情感分析（例如新浪微博），能针对每一条消息进行情感分类：正面或负面。该系统基于社交媒体，能够充分利用俚语，特殊词语等新新网络用语。请注意：该免费版本提供每天500条消息分类 - 超过此上限，将会被额外收费。
8. Viralheat Sentiment - Viralheat sentiment is free API and allows users to submit short chunks of text for sentiment scoring.
9. Text Processing - The WebKnox text processing API lets you process (natural) language texts. You can detect the text's language, the quality of the writing, find entity mentions, tag part-of-speech, extract dates, extract locations, or determine the sentiment of the text.
10. Skyttle  - Skyttle API is designed to turn any text into constituent terms (meaningful expressions), entities (names of people, place and things), and sentiment terms. Languages supported are English, Spanish, French, German, Chinese, Swedish, Greek, Czech, Italian and Russian.
11. Fluxifi NLP - Cloud based Natural Language Processing API. Includes Sentiment and Language Detection.
12. Sentiment Analysis Spanish - Sentiment analysis for Spanish language of any given tweet.
13. AlchemyAPI - AlchemyAPI provides advanced cloud-based and on-premise text analysis infrastructure that eliminates the expense and difficulty of integrating natural language processing systems into your application, service, or data processing pipeline.
14. nlpTools - Text processing framework to analyse Natural Language. It is especially focused on text classification and sentiment analysis of online news media (general-purpose, multiple topics).
15. Chinese Analytics - Soshio allows companies to quickly expand their understanding of the Chinese market. Its Chinese Analytics API provides Chinese text analytics and sentiment analysis capabilities for businesses to create their own social monitoring dashboard.
16. Truthy - Write scripts to work with our data, statistics, and images using the API. Download tweet volume over time, network layout, and statistics about memes and users, such as predicted political partisanship, sentiment score, language, and activity.
17. Speech2Topics - Yactraq Speech2Topics is a cloud service that converts audiovisual content into topic metadata via speech recognition & natural language processing. Customers use Yactraq metadata to target ads, build UX features like content search/discovery and mine Youtube videos for brand sentiment. In the past such services have been expensive and only used by large video publishers. The unique thing about Yactraq is we deliver our service at a price any product developer can afford.
18. Bitext Sentiment Analysis - The purpose of this service is to extract opinions from text. An opinion represents the subject an author is writing about and a sentiment score that classifies how positively or negatively the author feels towards that subject. Deep Linguistic Analysis is used to identify the subject the author is discussing.  
19. Textalytics Sentiment Analysis - Multilingual sentiment analysis of texts from different sources (blogs, social networks,…). Besides polarity at sentence and global level, Textalytics Sentiment Analysis 1.1 uses advanced natural language processing techniques to also detect the polarity associated to both entities and concepts in the text. Sentiment Analysis also gives the user the possibility of detecting the polarity of user-defined entities and concepts, making the service a flexible tool applicable to any kind of scenario.
20. Sentiment - This tool works by examining individual words and short sequences of words (n-grams) and comparing them with a probability model. The probability model is built on a prelabeled test set of IMDb movie reviews. It can also detect negations in phrases, i.e, the phrase "not bad" will be classified as positive despite having two individual words with a negative sentiment.
21. Starget sentiment analysis - his is a short text (a twitt or a single sentence) sentiment classification API. It has two types of analysis: one for finding more (but less accurate) sentiment snippets and another one for finding more accurate sentiment (but missing some difficult cases). 
22. Textalytics Media Analysis - Textalytics Media Analysis API analyzes mentions, topics, opinions and facts in all types of media. This API provides services for: - Sentiment analysis - Extracts positive and negative opinions according to the context.
23. Nevahold - Nevahold is a customer service application that leverages the social influence of its community to help users get their voice heard by companies. This API gives you real-time information of company's - Response Time on Facebook and Twitter - Average Response Rate - Customer Service Score - Sentiments - Geo Locations of interactions - Trending keywords
24. Free Natural Language Processing Service - 100% free service including sentiment analysis, content extraction, and language detection. Enjoy!

You should also check out our other useful API lists for machine learning , natural language processing , summarizing text , SMS APIs , and face recognition APIs .

Sent from Evernote

From Evernote:

We're on the cusp of deep learning for the masses. You can thank Google later — Tech News and Analysis Clipped from: http://gigaom.com/2013/08/16/were-on-the-cusp-of-deep-learning-for-the-masses-you-can-thank-google-later/

We're on the cusp of deep learning for the masses. You can thank Google later — Tech News and Analysis 

Google silently did something revolutionary on Thursday. It open sourced a tool called word2vec , prepackaged deep-learning software designed to understand the relationships between words with no human guidance. Just input a textual data set and let underlying predictive models get to work learning.
"This is a really, really, really big deal," said Jeremy Howard, president and chief scientist of data-science competition platform Kaggle. "… It's going to enable whole new classes of products that have never existed before." Think of Siri on steroids, for starters, or perhaps emulators that could mimic your writing style down to the tone.

When deep learning works, it works great 

To understand Howard's excitement, let's go back a few days. It was Monday and I was watching him give a presentation in Chicago about how deep learning was dominating the competition in Kaggle, the online platform where organization present vexing predictive problems and data scientists compete to create the best models. Whenever someone has used a deep learning model to tackle one of the challenges, he told the room, it has performed better than any model ever previously devised to tackle that specific problem.

Jeremy Howard (left) at Structure: Data 2012 (c) Pinar Ozger / http://www.pinarozger.com

But there's a catch: deep learning is really hard. So far, only a handful of teams in hundreds of Kaggle competitions have used it. Most of them have included Geoffrey Hinton or have been associated with him.
Hinton is a University of Toronto professor who pioneered the use of deep learning for image recognition and is now a distinguished engineer at Google, as well. What got Google really interested in Hinton — at least to the point where it hired him — was his work in an image-recognition competition called ImageNet . For years the contest's winners had been improving only incrementally on previous results, until Hinton and his team used deep learning to improve by an order of magnitude.

Neural networks: A way-simplified overview 

Deep learning, Howard explained, is essentially a bigger, badder take on the neural network models that have been around for some time. It's particularly useful for analyzing image, audio, text, genomic and other multidimensional data that doesn't lend itself well to traditional machine learning techniques.
Neural networks work by analyzing inputs (e.g., words or images) and recognizing the features that comprise them as well as how all those features relate to each other. With images, for example, a neural network model might recognize various formations of pixels or intensities of pixels as features.

A very simple neural network. Source: Wikipedia Commons

Trained against a set of labeled data, the output of a neural network might be the classification of an input as a dog or cat, for example. In cases where there is no labeled training data — a process called self-taught learning — neural networks can be used to identify the common features of their inputs and group similar inputs even though the models can't predict what they actually are. Like when Google researchers constructed neural networks that were able to recognize cats and human faces without having been trained to do so.

Stacking neural networks to do deep learning 

In deep learning, multiple neural networks are "stacked" on top of each other, or layered, in order to create models that are even better at prediction because each new layer learns from the ones before it. In Hinton's approach, each layer randomly omits features — a process called "dropout" — to minimize the chances the model will overfit itself to just the data upon which it was trained. That's a technical way of saying the model won't work as well when trying to analyze new data.
So dropout or similar techniques are critical to helping deep learning models understand the real causality between the inputs and the outputs, Howard explained during a call on Thursday. It's like looking at the same thing under the same lighting all the time versus looking at it in different lighting and from different angles. You'll see new aspects and won't see others, he said, "But the underlying structure is going to be the same each time."

An example of what features a neural network might learn from images. Source: Hinton et al

Still, it's difficult to create accurate models and to program them to run on the number of computing cores necessary to process them in a reasonable timeframe. It's also can be difficult to train them on enough data to guarantee accuracy in an unsupervised environment. That's why so much of the cutting-edge work in the field is still done by experts such as Hinton, Jeff Dean and Andrew Ng, all of whom had or still have strong ties to Google.
There are open source tools such as Theano and PyLearn2 that try to minimize the complexity, Howard told the audience on Monday, but a user-friendly, commercialized software package could be revolutionary. If data scientists in places outside Google could simply (a relative term if ever there was one) input their multidimensional data and train models to learn it, that could make other approaches to predictive modeling all but obsolete. It wouldn't be inconceivable, Howard noted, that a software package like this could emerge within the next year.

Enter word2vec 

Which brings us back to word2vec. Google calls it "an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words." Those "architectures" are two new natural-language processing techniques developed by Google researchers Tomas Mikolov, Ilya Sutskever, and Quoc Le (Google Fellow Jeff Dean was also involved, although modestly, he told me.) They're like neural networks, only simpler so they can be trained on larger data sets.
Kaggle's Howard calls word2vec the "crown jewel" of natural language processing. "It's the English language compressed down to a list of numbers," he said.
Word2vec is designed to run on a system as small as a single multicore machine (Google tested its underlying techniques over days across more than 100 cores on its data center servers). Its creators have shown how it can recognize the similarities among words (e.g., the countries in Europe) as well as how they're related to other words (e.g., countries and capitals). It's able to decipher analogical relationships (e.g., short is to shortest as big is to biggest), word classes (e.g., carnivore and cormorant both relate to animals) and "linguistic regularities" (e.g., "vector('king') – vector('man') + vector('woman') is close to vector('queen')).

Source: Google

Right now, the word2vec Google Code page notes, "The linearity of the vector operations seems to weakly hold also for the addition of several vectors, so it is possible to add several word or phrase vectors to form representation of short sentences."
This is accomplished by turning words into numbers that correlate with their characteristics, Howard said. Words that express positive sentiment, adjectives, nouns associated with sporting events — they'll all have certain numbers in common based on how they're used in the training data (so bigger data is better).

Smarter models means smarter apps 

If this is all too esoteric, think about these methods applied to auto-correct or word suggestions in text-messaging apps. Current methods for doing this might be as simple as suggesting words that are usually paired together, Howard explained, meaning a suggestion is could be based solely on the word immediately before it. Using deep-learning-based approaches, a texting app could take into account the entire sentence, for example, because the app would have a better understanding of what the all words really mean in context.
Maybe you could average out all the numbers in a tweet, Howard suggested, and get a vector output that would accurately infer the sentiment, subject and level of formality of the tweet. Really, the possibilities are limited only to the types of applications people can think up to take advantage of word2vec's deep understanding of natural language.

An example output file from word2vec that has grouped similar words

The big caveat, however, is researchers and industry data scientists still need to learn how to use word2vec. There hasn't been a lot of research done on how to best use these types of models, Howard said, and the thousands of researchers working on other methods of natural language processing aren't going to jump ship to Google's tools overnight. Still, he believes the community will come around and word2vec and its underlying techniques could make all other approaches to natural language processing obsolete.
And this is just the start. A year from now, Howard predicts, deep learning will have surpassed a whole class of algorithms in other fields (i.e., things other than speech recognition, image recognition and natural language processing), and a year after that it will be integrated into all sorts of software packages. The only questions — and they're admittedly big ones — is how smart deep learning models can get (and whether they'll run into another era of hardware constraints that graphical processing units helped resolve earlier this millennium) and how accessible software packages like word2vec can make deep learning even for relatively unsophisticated users.
"Maybe in 10 years' time," Howard proposed, "we'll get to that next level."

Data for dummies: 6 data-analysis tools anyone can use

Data for dummies: 6 data-analysis tools anyone can use:
If you care only about the cutting edge of machine learning and how to manage petabytes of big data, you might want to quit reading now and just come to our Structure:Data conference in March. But if you’re a normal person dealing with mere normal data, you’ll probably want to stick around. Although your data might not be that big or complex, that doesn’t mean it isn’t worth looking at in a new light.
With that in mind, here are six of the best free tools I’ve come across for helping we mere mortals analyze our data without having to know too much about, well, anything (I’d keep an eye on the still-under-wraps Datahero, too). I’ve gathered some personal data and tracked down some interesting public data sets to help demonstrate what a novice can do with them. Someone with more skills can certainly do a lot more, and larger datasets will provide greater statistical significance.

BigML

BigML is to machine learning what Blue Moon is to Belgian ales: a simple approach to something generally more complex — but also rather accessible and good enough to do the job in a pinch. I explained the service more thoroughly in recent post about it being used to generate predictions of Kickstarter success, but here’s how it works, in a nutshell: Users upload and format data (which is actually pretty easy), BigML discovers the myriad relationships between the variables and creates a predictive model, and users enter hypothetical data and receive a prediction.
I’m pretty bad when it comes to entering my data into Fitbit (see disclosure), but I was relatively good for a month this summer as I prepped for the Warrior Dash, and that’s the data I used to demonstrate BigML. This prediction of how many calories I can expect to burn in a day would work a lot better if I had a bigger sample size and hadn’t occasionally forgotten to log calories and hours slept, but you get the point. The first image is the model the service generated; the second is the prediction interface.

Google Fusion Tables

The user interface for Google Fusion Tables  isn’t what I’d call pretty (“sparse” is probably a better description), but the still-in-experimental-mode visualization tool sure is easy if your data is nicely formatted. I created this interactive map simply by uploading a publicly available dataset about gun violence and clicking the button to create a map:

For this simple comparison of gun ownership and gun homicide rates, I just checked the countries by which I wanted to filter the chart. Easy.:

Infogram

If you have really simple data — like a few columns and a handful of rows — Infogram might be the easiest to use of the bunch. The company launched last year with a variety of infographic templates, but it has since expanded to include a large number of charts and graphs, too (including line, pie, pictorial, treemap and bubble). Furthermore, it gives sample data, which you can use as an example to enter your own or format the table you want to upload, and the interactive charts embed nicely into web pages (ours, at least).
Here are the top 10 things I ate during the time I was logging food via Fitbit, excluding copious amounts of beer, water, coffee and Diet Pepsi that I didn’t record.

What I ate | Create infographics

In July, I made this chart with Infogram comparing infrastructure spending trends among internet companies.

Who spent what on infrastructure | Create infographics

And here’s a sample of the simplest chart in the world.

I am this far through my to-do list | Create infographics

Many Eyes

Many Eyes is a free web service run by IBM that includes a wide variety of visualizations ranging from maps to pie charts to scatter plots. But what makes it stand apart from the others is the suite of text-analysis tools it offers — not only are they fairly novel, but all they require users to do is paste a page of plain text into the web interface and press a button to visualize it. I used it to analyze the last 15 posts I’ve written for GigaOM.
What did I find? For starters, I use the words “data,” “Facebook” and “users” a lot.

When it comes to two-word combinations, “big data,” “data centers” and “hard drives” are among the biggies.

This one is particularly interesting, showing how I tend to form phrases around certain words with common conjunctions, or just a space, in between.

Apparently, out of 10,013 words, I only used “cloud” 20 times. I usually followed it up with “provider,” “servers,” “computing,” “-based” and “providers.”

For fun, I also made a word cloud based on couple month’s worth of Fitbit food logs. It turns out, you can take the boy out of Wisconsin, but …

Statwing

Statwing might be my favorite of the bunch, if only because it’s so simple yet actually tries to teach users about statistics. You upload data, check the variables you’re concerned with, and it plots their relationship. (It also can describe the variables by highlighting the sample size, minimum, maximum, mean, median and standard deviation.) Graphs are accompanied by explanations as to how strong the correlation is based on various statistical metrics, as well as the results of a linear regression model.
To demonstrate Statwing, I went back to the Fitbit data. Of the variables that Fitbit tracks, some correlations are easy to predict (e.g., steps and calories burned), but I was kind of surprised to see that the 86 minutes a day I spent being fairly active really weren’t that good of an expenditure of my time.

Tableau Public

Tableau Public, the only free version of the popular business-intelligence software, was clearly designed with business users in mind. It expects a lot of structure in the data, and although you can edit almost every aspect of it within the application to get it into usable shape, the service doesn’t allow much guidance if you don’t speak the language of BI (it also requires Windows). But the software is very good at deciphering the characteristics of different variables, the drag-and-drop operation makes it kind of easy to experiment and the wide array of visualizations look really nice.
Using my Fitbit data (and here’s where you see how lax I am at data entry), I created a line graph comparing the calories I ate each day with the calories I burned. Assuming I didn’t go crazy eating on the days I forgot to make entries, the good news is I never ate more calories than I burned. (Note: Although these are static images, Tableau Public actually lets you embed interactive charts, which I’ve used in the past on several occasions, but they don’t always fit well within our pages.)
Here’s one I played around with a while back charting Amazon’s “Other” revenue againt the number of objects stored in Amazon S3.
Finally, here is my first-ever (I think) Tableau chart, which uses the raw data on government takedown requests that Google provided along with its Transparency Report in October 2011. You can read that post and play with the interactive version here.


There is, however, one disclaimer that applies to all of these tools: I didn’t get into cleaning and formatting data, which can be a somewhat arduous process. Many tools expect some sort of structure to the data — the X axis to be in columns and the Y axis in rows, measurements without units (e.g., grams), etc. — that just isn’t present if you’re downloading an Excel or CSV file rather than creating it yourself. Sometimes, with comprehensive datasets like your Fitbit Premium data, you’ll have to separate or combine the relevant data into new spreadsheet tables before uploading it to a service. But once you have the data ready to go, these tools can help you analyze it, visualize it and hopefully glean some insights from it.
Disclosure: Fitbit is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, founder of Giga Omni Media, is also a venture partner at True.



Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The importance of putting the U and I in visualization
• Health care and big data in 2012
• Cloud computing infrastructure: 2012 and beyond

Statistics about Hadoop and Mapreduce Algorithm Papers

Statistics about Hadoop and Mapreduce Algorithm Papers:

Underneath are statistics about which 20 papers (of about 80 papers) were most read in our 3 previous postings about mapreduce and hadoop algorithms (the postings have been read approximately 5000 times). The list is ordered by decreasing reading frequency, i.e. most popular at spot 1.

1. MapReduce-Based Pattern Finding Algorithm Applied in Motif Detection for Prescription Compatibility Network
   authors: Yang Liu, Xiaohong Jiang, Huajun Chen , Jun Ma and Xiangyu Zhang – Zhejiang University
2. Data-intensive text processing with Mapreduce
   authors: Jimmy Lin and Chris Dyer – University of Maryland
3. Large-Scale Behavioral Targeting
   authors: Ye Chen (eBay), Dmitry Pavlov (Yandex Labs) and John F. Canny (University of California, Berkeley)
4. Improving Ad Relevance in Sponsored Search
   authors: Dustin Hillard, Stefan Schroedl, Eren Manavoglu, Hema Raghavan and Chris Leggetter (Yahoo Labs)
5. Experiences on Processing Spatial Data with MapReduce
   authors: Ariel Cary, Zhengguo Sun, Vagelis Hristidis and Naphtali Rishe – Florida International University
6. Extracting user profiles from large scale data
   authors: Michal Shmueli-Scheuer, Haggai Roitman, David Carmel, Yosi Mass and David Konopnicki – IBM Research, Haifa
7. Predicting the Click-Through Rate for Rare/New Ads
   authors: Kushal Dave and Vasudeva Varma – IIIT Hyderabad
8. Parallel K-Means Clustering Based on MapReduce
   authors: Weizhong Zhao, Huifang Ma and Qing He – Chinese Academy of Sciences
9. Storage and Retrieval of Large RDF Graph Using Hadoop and MapReduce
   authors: Mohammad Farhan Husain, Pankil Doshi, Latifur Khan and Bhavani Thuraisingham – University of Texas at Dallas
10. Map-Reduce Meets Wider Varieties of Applications
    authors: Shimin Chen and Steven W. Schlosser – Intel Research
11. LogMaster: Mining Event Correlations in Logs of Large-scale Cluster Systems
    authors: Wei Zhou, Jianfeng Zhan, Dan Meng (Chinese Academy of Sciences), Dongyan Xu (Purdue University) and Zhihong Zhang (China Mobile Research)
12. Efficient Clustering of Web-Derived Data Sets
    authors: Luıs Sarmento, Eugenio Oliveira (University of Porto), Alexander P. Kehlenbeck (Google), Lyle Ungar (University of Pennsylvania)
13. A novel approach to multiple sequence alignment using hadoop data grids
    authors: G. Sudha Sadasivam and G. Baktavatchalam – PSG College of Technology
14. Web-Scale Distributional Similarity and Entity Set Expansion
    authors: Patrick Pantel, Eric Crestan, Ana-Maria Popescu, Vishnu Vyas (Yahoo Labs) and Arkady Borkovsky (Yandex Labs)
15. Grammar based statistical MT on Hadoop
    authors: Ashish Venugopal and Andreas Zollmann (Carnegie Mellon University)
16. Distributed Algorithms for Topic Models
    authors: David Newman, Arthur Asuncion, Padhraic Smyth and Max Welling – University of California, Irvine
17. Parallel algorithms for mining large-scale rich-media data
    authors: Edward Y. Chang, Hongjie Bai and Kaihua Zhu – Google Research
18. Learning Influence Probabilities In Social Networks
    authors: Amit Goyal, Laks V. S. Lakshmanan (University of British Columbia) and Francesco Bonchi (Yahoo! Research)
19. MrsRF: an efficient MapReduce algorithm for analyzing large collections of evolutionary trees
    authors: Suzanne J Matthews and Tiffani L Williams – Texas A&M University
20. User-Based Collaborative-Filtering Recommendation Algorithms on Hadoop
    authors: Zhi-Dan Zhao and Ming-sheng Shang

CS345A: Data Mining

http://infolab.stanford.edu/~ullman/mining/2009/index.html

Date	Topic	PowerPoint Slides	PDF Document
1/7	Introductory Remarks (JDU)	PPT	PDF
1/7	Introductory Remarks (AR)	PPT	PDF
1/12	Map-Reduce	PPT	PDF
1/14	Frequent Itemsets 1	PPT	PDF
1/14-1/21	Frequent Itemsets 2	PPT	PDF
1/16	Peter Pawlowski's Talk on Aster Data	PPTX	PDF
1/16	Nanda Kishore's Talk on ShareThis	PPT	PDF
1/26	Recommendation Systems	PPT	PDF
1/28	Shingling, Minhashing, Locality-Sensitive Hashing	PPT	PDF
2/2	Applications and Variants of LSH	PPT	PDF
2/2-2/4	Distance Measures, Generalizations of Minhashing and LSH	PPT	PDF
2/4	High-Similarity Algorithms	PPT	PDF
2/9	PageRank	PPT	PDF
2/11	Link Spam, Hubs & Authorities	PPT	PDF
2/18	Generalization of Map-Reduce	PPT	PDF
2/18-2/23	Clustering	PPT	PDF
2/23	Streaming Data	PPT	PDF
2/25	Relation Extraction	PPT	PDF
3/2	On-Line Algorithms, Advertising Optimization	PPT	PDF
3/4	Algorithms on Streams	PPT	PDF