What is Big Data?

What is Big Data? How do you learn big data?
What is big data and what is its use?

Goals and Objective :

1. What is Big Data?
2. Parameters of Big Data
3. Traditional Solutions
4. Challenges of Big Data
5. Hadoop -A Good Solution
6. Hadoop Vs RDBMS

What is Big Data?

Big Data is Data in a huge volume that is unprocessed. We as a customer or a user of a particular service generate lots of data every single second. We need storage space to store that information and computation power to process that data.

If I have generated 2 Terabytes of data then do I call it a Big Data problem?

The answer is No.

So, What are some parameters that help us to categorize whether a particular problem is a big data problem or not?

Parameters are 3V’s:

1. Volume: Data got generated in a huge amount that is more than Terabytes of data.

The volume of Big Data

2. Velocity: Data got generated at high speed. For example, NASA is generating more than 2 Gigabytes of data every hour.

High-Velocity Data

3. Variety: Data consists of a huge variety. For example, Music, Video, Text, etc, are different varieties of data.

Variety of Big Data

Challenges of Big Data :

Now we know that people are generating lots of data so this big data comes with lots of challenges :

1. Storage: Storage plays a main important role, as we are generating more than Terabytes of data but where to store that data? We need a database to store this huge amount of data.
2. Computational Efficiency: Huge amount of data requires lots of processing capabilities which eventually get decreasing by increasing data.
3. Data Loss: Due to the corruption of physical device or hardware failure, data loss occurs. That is we need to have some important recovery strategies to recover the data.
4. Cost: With the increase in data and computation capabilities it will eventually increase the cost of both the challenges. So we want cost-effective strategies to deal with these challenges.

Traditional Solutions

Traditionally we are using MySQL, Oracle to structure and store the data effectively. But there are limitations in terms of Scalability because when it comes to increasing the volume of data then the time to denormalize the data becomes a challenge.

Another problem with the traditional method is that we can only store structured data that means the traditional database will struggle to store structured data.

Hadoop -A Good Solution

In order to rectify the limitations of traditional methods, Hadoop comes into play.

Hadoop is an open-source distributed computing framework that manages :

1. Support Huge Volume
2. Storage Efficiency
3. Good Data Recovery Solution
4. Horizontal Scaling
5. Cost-Effective
6. Easy for Programmers and Non-Programmers

So with these types of features of Hadoop Can we replace RDBMS?

The answer is No because Hadoop and RDBMS are having there own advantages and disadvantages.

Hadoop Vs RDBMS

1. Hadoop works on Dynamic Schema whereas RDBMS works on Static Schema.
2. Hadoop works on Linear Scale whereas RDBMS works on the Non-Linear Scale.
3. Hadoop works on more than Petabytes of data whereas RDBMS works on Gigabytes of data.
4. Hadoop works on Batch processing whereas RDBMS works on Interactive and Batch processing.
5. Hadoop works on ‘Write Once, Read Many Times’ whereas RDBMS works on ‘Read Write Many Times’.