Big Data Analytics

A year ago, I had to start a POC on Hadoop and I had no idea about what Hadoop is.

I would explain the way I started with and which helped others as well.

1. Go through some introductory videos on Hadoop
Its very important to have some high level idea of hadoop before directly starting working on it. These introductory videos will help in understanding the scope of Hadoop and the use cases where it can be applied. There are a lot of resources available online for the same and going through any of the videos will be beneficial.

2. Understanding MapReduce
The second thing which helped me was to understand what Map Reduce is and how it works. It is explained very nicely in this paper: http://static.googleusercontent....

Another nice tutorial is available here : http://ksat.me/map-reduce-a-real...

3. Getting started with Cloudera VM
Once you understand the basics of Hadoop, you can download the VM provided by cloudera and starting running some hadoop commands on it. You can download the VM from this link: http://www.cloudera.com/content/...

It would be nice to get familiar with basic Hadoop commands on the VM and understanding how it works.

4. Setting up the standalone/Pseudo distributed Hadoop
I would recommend setting up your own standalone Hadoop on your machine once you are familiar with Hadoop using the VM. The steps for installing are explained very nicely on this blog by Michael G. Noll : Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll

5. Understanding the Hadoop Ecosystem
It would be nice to get familiar with other components in the Hadoop ecosystem like Apache Pig, Hive, Hbase, Flume-NG, Hue etc. All these serve different purposes and having some information on all these will be really helpful in building any product around the hadoop ecosystem. You can install all these easily on your machine and get started with them. Cloudera VM by has most of these installed already.

6. Writing Map Reduce Jobs
Once you are done with steps 1-5, I don't think writing Map Reduce would be a challenge. It is explained thoroughly in The Definitive Guide. If MapReduce really interests you a lot, I would suggest reading this book Mining Massive Datasets by Anand Rajaraman, Jure Leskovec and Jeffrey D. Ullman : Page on Stanford


I might have missed some points here but that is how I started with Hadoop and my journey so far has been really interesting and rewarding.

Where can I find huge data sets, is a question in front of every person who is aiming to develop, test or study big data analytics tools. Here we provide the list of data sets available in public domain.

More Than 1 TB data

1000 Genomes
This project makes 260TB of human genome data available to the users. 

Internet Archive
More than 80TB of data is available for research of internet web crawling

ClueWeb09 
The TREC conference provided this data set to public. You will need to sign agreement and pay fee of (upto $610) to cover sneakernet data transfer. Total 5TB data is available. 
http://lemurproject.org/clueweb09.php/

ClueWeb12  
ClueWeb12 is available as freebase annotation.

CNetS 
Indiana University makes a 2.5 TB click dataset available 

ICWSM 
ICWSM have made their 2011 conference blog posts available to public. You will have to register an actual form, but this is available for free, which is 2.1TB in compressed format. 

Proteome Commons 
Proteome commons have made several large datasets available. The largest, the Personal Genome Project, is 1.1 TB in size. There are several others over 100 GB in size.




Reference Energy Disaggregation Data Set 
The has data on home energy usage it's about 500 GB compressed.

Tiny Images 
This dataset has 227 GB of image data and 57 GB of metadata.

ImageNet 
The dataset is pretty big.

MOBIO 
The dataset is about 135 GB of video and audio data

Yahoo! Webscope
The  program makes several 1 GB+ datasets available to academic researchers, including an 83 GB data set of Flickr image features and the dataset used for the 2011 KDD Cup, from Yahoo! Music, which is a bit over 1 GB.

Google 
made a dataset mapping words to Wikipedia URLs (i.e., concepts) [15]. The dataset is about 10 GB compressed.

Yandex 
has recently made a very large web search click dataset available [1]. You'll have to register online for the contest to download. It's about 5.6 GB compressed.

Freebase 
Freebase provides data dumps on regular basis. The largest is their Quad dump, which is about 3.6 GB compressed.

Open American National Corpus
This is about 4.8 GB uncompressed.

Wikipedia 
made a dataset containing information about edits available for a recent Kaggle competition. The training dataset is about 2.0 GB uncompressed.

Research and Innovative Technology Administration (RITA)
RITA has made available a dataset about the on-time performance of domestic flights operated by large carriers. The ASA compressed this dataset and makes it available for download.

Wiki-links 
data made available by Google is about 1.75 GB total [20].

Amazon Public datasets
Amazon Provides centralized repository of public data sets available on Amazon cloud to integrate & downlaod.

Huge Elephant representing as a big data

Big Data:

Big data a new buzzword in a market we are hearing all around now a days. What really big data means?

In simple words big data says, data sets which are so large, complex and unstructured that makes them difficult and impossible to process, analyze using traditional databases.

Now a days people are creating huge data every day. Each organization is generating huge pile of data which is very unstructured and huge which makes it difficult to process and analyze. Big data provides us a way to manage these huge data sets.

How much data is really a big data?

Some GBs of data, some TBs of data or some petabytes of data?

Well this is a debatable questions as to what we should call a really big data. Ideally data which is complex and more than hundreds of GB's which is difficult to process with traditional ways should be called as a big data. It scales up to terabytes to petabytes and more.

Big Data Analysis:

Analyzing these huge data sets which forms a complex and unstructured data is difficult with traditional way. This is where big data analytics comes in to picture.

We got to know what the big data is, but how to process these huge data sets and analyse them to generate reports and specific trends from huge datasets?

Implementing Big Data:
There are various open source technologies used to analyse these huge data sets.
Popular are Hadoop, Pig, Hive. No SQL dabases lik Hbase, Cassandra, in memory databases like Redis, R programming etc.

Will Big Data be a future:

This will be a point to be seen in near future, as of now every one is talking about big data. It is this and that, we can process huge data etc etc and people are exploring it. But very few people know how to use this big data technologies, every one is following others but knowing very less about how it help us.

Watch out this space.