{"id":949,"date":"2017-11-29T14:06:51","date_gmt":"2017-11-29T14:06:51","guid":{"rendered":"http:\/\/blog.cloudxlab.com\/?p=949"},"modified":"2019-01-08T12:45:47","modified_gmt":"2019-01-08T12:45:47","slug":"big-data-introduction","status":"publish","type":"post","link":"https:\/\/cloudxlab.com\/blog\/big-data-introduction\/","title":{"rendered":"What is Big Data? An Easy Introduction to Big Data Terminologies"},"content":{"rendered":"

Unless you’ve been living under the rock, you must have heard or read the term – Big Data. But many people don’t know what Big Data actually means. Even if they do then the definition of the same is not clear to them. If you’re one of them then don’t be disheartened. By the time you complete reading this very article, you will have a clear idea about Big Data and its terminology. <\/span><\/p>\n

What is Big Data?<\/h2>\n

In very simple words, Big Data is data of very big size which can not be processed with usual tools like file systems & relational databases. And to process such data we need to have distributed architecture. In other words, we need multiple systems to process the data to achieve a common goal.<\/span><\/p>\n

<\/p>\n

Generally, we classify the problems related to the handling of Big Data into three buckets:<\/p>\n

\"Characteristics<\/p>\n

1. Volume<\/h3>\n

When the problem we are solving is related to how we would store such huge data, we call it Volume. For example, Facebook stores 600 TB of Data in just one day!<\/span><\/p>\n

2. Velocity<\/h3>\n

When we are trying to handle many requests per second, we call this characteristic Velocity. For example, the number of requests received by Facebook or Google per second.<\/span><\/p>\n

3. Veracity<\/h3>\n

If the problem at hand is complex or data that we are processing is complex, we call such problems as related to variety. For example, problems involving complex data structures like Maps & Social Graphs.<\/span><\/p>\n

Data could be termed as Big Data if either Volume, Velocity or Variety becomes impossible to handle using traditional tools.<\/span><\/p>\n

Why do we need Big Data now?<\/h2>\n

You will see the answer to this question when we look at the huge transition from Analog storage to Digital storage.<\/span><\/p>\n

For your information, Paper, Tapes etc are examples of analog storage while CDs, DVDs, hard disk drives are considered digital storage.<\/span><\/p>\n

\"Global<\/p>\n

This graph shows that the digital storage has started increasing exponentially after 2002 while analog storage remained practically same.<\/span><\/p>\n

The year 2002 is called beginning of the digital age. Why so? The answer is twofold: Devices<\/em> & Connectivity<\/em>. Devices became cheaper, faster and smaller and on the other hand, the connectivity improved.\u00a0<\/span><\/p>\n

This lead to a lot of very useful applications such as a very vibrant world wide web, social networks, and Internet of things leading to huge data generation.<\/span><\/p>\n

With the huge data generation, it became practically impossible to store & process such humongous data. Let’s go through some basics to better understand the need for multiple systems to process big data.<\/span><\/p>\n

Roughly, the computer is made of 4 components.<\/span><\/p>\n

1. CPU<\/strong> – Which executes instructions. CPU is characterized by its speed. More the number of instructions it can execute per second, faster it is considered.<\/span><\/p>\n

2. RAM<\/strong> – Random access memory. While processing, we load data into RAM. If we can load more data into ram, CPU can perform better. So, RAM has two main attributes which matter: Size and its speed of reading and writing.<\/span><\/p>\n

3. Storage Disk<\/strong> – To permanently store data, we need hard disk drive or solid-state drive. The SSD is faster but smaller and costlier. The faster and bigger the disk, faster we can process data.<\/span><\/p>\n

4. Network<\/strong> – Another component that we frequently forget while thinking about the speed of computation is the network. Why? Often our data is stored on different machines and we need to read it over a network to process.<\/span><\/p>\n

While processing Big Data at least one of these four components become the bottleneck. In fact, all of the following components can impact the speed of computing – CPU, Memory Size, Memory Read Speed, Disk Speed, Disk Size, and Network Speed. <\/span><\/p>\n

Which is why we need to move to multiple computers or distributed computing architecture.<\/span><\/p>\n

Big Data Applications<\/h2>\n

So far we have tried to establish that while handling humongous data we would need a new set of tools which can operate in a distributed fashion.
\nBut who would be generating such data or who would need to process such humongous data? A quick answer is everyone.<\/span><\/p>\n

Now, let us try to take few examples.<\/span><\/p>\n

1. E-Commerce Recommendation<\/h3>\n

In e-commerce industry, the recommendation is a great example of Big Data processing. The recommendation, also known as collaborative filtering is the process of suggesting someone a product based on their preferences or behavior.<\/span><\/p>\n

The e-commerce website would gather a lot of data about the customer’s behavior. In a very simplistic algorithm, we would basically try to find similar users and then cross-suggest them the products. So, more the users, better the results.<\/span><\/p>\n

As per Amazon, a major chunk of their sales happens via recommendations on the website and emails. <\/span><\/p>\n

As of today, generating recommendations have become pretty simple. The engines such as MLLib or Mahout have made it very simple to generate recommendations on humongous data. All you have to do is format the data in the three column format: user id, movie id, and ratings.<\/span><\/p>\n

2. A\/B Testing<\/h3>\n

A\/B Testing is a process to compare the response of the users with respect to two different variations.<\/span><\/p>\n

\"Big<\/span><\/p>\n


\nAs you can see in the diagram, randomly selected half of the users are shown variation A and other half is shown variation B. We can clearly see that variation A is very effective because it is giving double conversions.<\/span><\/p>\n

This method is effective only if we have a significant amount of users. Also, the ratio of the users need not be 50-50.<\/span><\/p>\n

To manage so many variations on such a high number of users, we generally need Big Data platforms.<\/span><\/p>\n

Big Data Customers<\/h2>\n

1. Government<\/h3>\n

Since governments have huge data about the citizens, any analysis would be Big Data analysis. The applications are many.<\/span><\/p>\n

First is Fraud Detection<\/em>. Be it antimony laundering or user identification, the amount of data processing required is really high. <\/span><\/p>\n

In Cyber Security Welfare and Justice, the Big Data is being generated and Big Data tools are getting adopted.<\/span><\/p>\n

2. Telecom<\/h3>\n

The telecom companies can use big data in order to understand why their customers are leaving and how they can prevent their customers from leaving. This is known as customer churn prevention<\/em>. <\/span><\/p>\n

The data that could help in customer churn prevention is<\/span><\/p>\n