Big Data Analytics

A year ago, I had to start a POC on Hadoop and I had no idea about what Hadoop is.

I would explain the way I started with and which helped others as well.

1. Go through some introductory videos on Hadoop
Its very important to have some high level idea of hadoop before directly starting working on it. These introductory videos will help in understanding the scope of Hadoop and the use cases where it can be applied. There are a lot of resources available online for the same and going through any of the videos will be beneficial.

2. Understanding MapReduce
The second thing which helped me was to understand what Map Reduce is and how it works. It is explained very nicely in this paper: http://static.googleusercontent....

Another nice tutorial is available here : http://ksat.me/map-reduce-a-real...

3. Getting started with Cloudera VM
Once you understand the basics of Hadoop, you can download the VM provided by cloudera and starting running some hadoop commands on it. You can download the VM from this link: http://www.cloudera.com/content/...

It would be nice to get familiar with basic Hadoop commands on the VM and understanding how it works.

4. Setting up the standalone/Pseudo distributed Hadoop
I would recommend setting up your own standalone Hadoop on your machine once you are familiar with Hadoop using the VM. The steps for installing are explained very nicely on this blog by Michael G. Noll : Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll

5. Understanding the Hadoop Ecosystem
It would be nice to get familiar with other components in the Hadoop ecosystem like Apache Pig, Hive, Hbase, Flume-NG, Hue etc. All these serve different purposes and having some information on all these will be really helpful in building any product around the hadoop ecosystem. You can install all these easily on your machine and get started with them. Cloudera VM by has most of these installed already.

6. Writing Map Reduce Jobs
Once you are done with steps 1-5, I don't think writing Map Reduce would be a challenge. It is explained thoroughly in The Definitive Guide. If MapReduce really interests you a lot, I would suggest reading this book Mining Massive Datasets by Anand Rajaraman, Jure Leskovec and Jeffrey D. Ullman : Page on Stanford


I might have missed some points here but that is how I started with Hadoop and my journey so far has been really interesting and rewarding.

Big Data and Big data analytics:

Analytics: This is a technique, deriving insights from data

Big Data: Huge data sets, which are difficult to analyze in traditional ways.

Huge Elephant representing as a big data

Big Data:

Big data a new buzzword in a market we are hearing all around now a days. What really big data means?

In simple words big data says, data sets which are so large, complex and unstructured that makes them difficult and impossible to process, analyze using traditional databases.

Now a days people are creating huge data every day. Each organization is generating huge pile of data which is very unstructured and huge which makes it difficult to process and analyze. Big data provides us a way to manage these huge data sets.

How much data is really a big data?

Some GBs of data, some TBs of data or some petabytes of data?

Well this is a debatable questions as to what we should call a really big data. Ideally data which is complex and more than hundreds of GB's which is difficult to process with traditional ways should be called as a big data. It scales up to terabytes to petabytes and more.

Big Data Analysis:

Analyzing these huge data sets which forms a complex and unstructured data is difficult with traditional way. This is where big data analytics comes in to picture.

We got to know what the big data is, but how to process these huge data sets and analyse them to generate reports and specific trends from huge datasets?

Implementing Big Data:
There are various open source technologies used to analyse these huge data sets.
Popular are Hadoop, Pig, Hive. No SQL dabases lik Hbase, Cassandra, in memory databases like Redis, R programming etc.

Will Big Data be a future:

This will be a point to be seen in near future, as of now every one is talking about big data. It is this and that, we can process huge data etc etc and people are exploring it. But very few people know how to use this big data technologies, every one is following others but knowing very less about how it help us.

Watch out this space.