E&ICT IIT Roorkee - Data Engineering Course at Rs. 1,999 (90% OFF) | Expires in

 Enroll Now
E&ICT

Electronics & ICT Academy

Indian Institute Of Technology Roorkee

An initiative of Ministry of Electronics and Information Technology (MeitY) Govt. of India


Faculty Development Program in Big Data with Spark

Certification program from E & ICT Academy IIT Roorkee in Big Data with Spark. Learn Spark, Spark RDD, Spark Streaming, Kafka, SparkR, SparkSQL, MLlib, and GraphX From Industry Experts.

178 Ratings       566 learners

  30+ hours training

  90 days of Lab

  24x7 Support

Projects

  Compatible with Hortonworks, Cloudera Certifications


The E&ICT Academy project is sponsored by Ministry of Electronics and Information Technology, Govt. of India. The E&ICT courses lay special emphasis on hands-on learning with participation from industry in the emerging areas of E&ICT domain. Our programs enable the participants and institutes to build industry connects, upgrade lab facilities and create opportunities for collaboration. As of now we have conducted 70 courses and trained over 3000 beneficiaries successfully. Learn More

E&ICT Academy IIT Roorkee supported by Ministry of Electronics and Information Technology ( MeitY) in collaboration with CloudxLab, is conducting a training program in Big Data with Spark.

As humans, we are immersed in data in our every-day lives. As per IBM, the data doubles every two years on this planet. The value that data holds can only be understood when we can start to identify patterns and trends in the data. Normal computing principles do not work when data becomes huge.

There is massive growth in the big data space, and job opportunities are skyrocketing, making this the perfect time to launch your career in this space.

In this course, you will learn Spark to drive better business decisions and solve real-world problems.

Eligibiliy criteria - Any professional, faculty of any Government or private institutions, employees of government.



1 course

Learn from industry experts. Follow the suggested order or choose your own.

Projects & Lab

Apply the skills you learn on a distributed cluster to solve real-world problems.

Certificate

Certification from E&ICT Academy IIT Roorkee

Best-in-class Support

24×7 support and forum access to answer all your queries throughout your learning journey.

Certifications

Compatible to Hortonworks Certified Developer (HDPCD): Spark
Enrollment

Faculty Development Program

31 May
Fri, Sat
(5 weeks)
9:30 p.m. - 12:30 a.m. America/New_York

90 days lab
15 536
7 Jun
Fri, Sat
(5 weeks)
9:30 p.m. - 12:30 a.m. America/New_York

90 days lab
359 536
Learning Path
Download Course Syllabus

Course

About the Course


What is Big Data?

Why Now?

Big Data Use Cases

Various Solutions

Spark Ecosystem Walkthrough

Quiz

Understanding the CloudxLab

CloudxLab Hands-On

Spark Hands-on

Quiz and Assessment

Basics of Linux - Quick Hands-On

Understanding Regular Expressions

Quiz and Assessment

Setting up VM (optional)

As part of this session we will do a recap of the sessions on Hadoop Distributed File System(HDFS) and Yet Another Resource Negotiator (YARN).

This is needed because most of the spark applications use data from HDFS and in most of deployments, spark applications are run on YARN clusters.

Introduction to Scala?

Accessing Scala using CloudxLab

Getting Started: Interactive, Compilation, SBT

Types, Variables & Values

Functions

Collections

Classes

Parameters

More Features

Quiz and Assessment

What is Apache Spark?

Why Spark?

Using the Spark Shell and various ways of running spark on CloudxLab

Example 1 - Performing Word Count

Understanding Spark Cluster Modes on YARN

RDDs (Resilient Distributed Datasets)

General RDD Operations: Transformations & Actions

RDD lineage

RDD Persistence Overview

Distributed Persistence

Learn operations on Key-Value Based RDD

Solving various problems using RDD

Creating the SparkContext

Building a Spark Application (Scala, Java, Python)

The Spark Application Web UI

Configuring Spark Properties

Running Spark on Cluster

RDD Partitions

Executing Parallel Operations

Stages and Tasks

Project: Churning the logs of NASA Kennedy Space Center WWW server

Using Accumulators & Creating Custom Accumulators

Using Broadcast variables

We will learn key performance considerations:

  1. Level of Parallelism
  2. Serialization Format
  3. Memory Management
  4. Hardware Provisioning

Understanding Caching & Persistence

We will Data Partitioning/Re-partitioning techniques.

A project to consider the above optimization techniques.

We will how to create custom partitioner.

Understand the Spark Runtime Architecture and various components such as Driver, Executor, Cluster Manager etc.

Learn what goes inside when we launch an spark application.

We will learn the two modes of Spark: Local and Cluster.

How to launch a program on YARN, AWS Cluster etc.

How to setup spark in standalone mode.

Understand and demonstrate on how to run drive in various modes.

Learn how to package the dependencies of your code.

Understand how to use the Spark-Submit and various command line options.

Common Spark Use Cases

Example 1 - Data Cleaning (Movielens)

Example 2 - Understanding Spark Streaming

Understanding Kafka

Example 3 - Spark Streaming from Kafka

Iterative Algorithms in Spark

Project: Real-time analytics of orders in an e-commerce company

Spark SQL and the SQL Context

Creating DataFrames

Transforming and Querying DataFrames

Saving DataFrames

Solving problems with DataFrames and RDDs

Comparing Spark SQL, Impala, and Hive-on-Spark

Understanding and loading various Input formats: JSON, XML, AVRO, SequenceFile?, Parquet, Protocol Buffers.

Comparing Compressions

Understanding Row Oriented and Column Oriented Formats - RCFile?

Understanding Machine Learning

MlLib Example: Recommendations on movie lense data

Understanding various Packages of MLlib

SparkR Example.

Basics of Graph Processing: Covers the understanding of what does it mean by graph processing in real life with examples. What are other frameworks providing graph computing?

GraphX Overview: What is GraphX? Understanding the functionalities and algorithms provided by GraphX. And how does GraphX work. Along with comparision with other similar products.

Implementing Page rank using GraphX: We will learn the basics of PageRank - the algorithm that made Google. The we learn how to implement using GraphX.

Projects

Projects

1. Generate movie recommendations using Spark MLlib


2. Derive the importance of various handles at Twitter using Spark GraphX


3. Churn the logs of NASA Kennedy Space Center WWW server using Spark to find out useful business and devops metrics


4. Write end-to-end Spark application starting from writing code on your local machine to deploying to the cluster


5. Build real-time analytics dashboard for an e-commerce company using Apache Spark, Kafka, Spark Streaming, Node.js, Socket.IO and Highcharts

Certificate

Certificate

Earn your certificate from E&ICT Academy IIT Roorkee.

Get a joint, verifiable certification from IIT Roorkee and CloudxLab after the completion any of the FDP program.


Differentiate yourself

The knowledge you have gained from working on projects, videos, quizzes, hands-on assessments and case studies gives you a competitive edge.


Share your achievement

Highlight your new skills on your resume, LinkedIn, Facebook and Twitter. Tell your friends and colleagues about it.

 Course Certificate Sample
Course Creators
Sandeep Giri

Sandeep Giri

Founder at CloudxLab
Past: Amazon, InMobi, D.E.Shaw
Course Developer
Sandeep Giri

Sanjeev Manhas

Associate Professor,
IIT Roorkee
Course Advisor
Sandeep Giri

R. Balasubramanian

Professor,
IIT Roorkee
Course Developer
Sandeep Giri

Partha Pratim Roy

Assistant Professor,
IIT Roorkee
Course Developer
Abhinav Singh

Abhinav Singh

Co-Founder at CloudxLab
Past: Byjus
Course Developer
 Jatin Shah

Jatin Shah

Ex-LinkedIn, Yahoo, Yale CS Ph.D.
IIT-B
Course Advisor

Reviews

(4.9 out of 5)
...

I have started learning 3 months ago and I really gained much info and practical experience. I completed the “Big Data with Spark” course and the learning journey really exceeded my expectations.

The course structure and topics were great, well organized and comprehensive, even the basics of Linux were covered in a very simple way. There were always exercises and hands-on that build better understanding, also the lab environment and provided online tools were great help and let you practice everything without having to install anything on your PC except the web browser.

In addition, for the live sessions, it was really a joy attending them each weekend, our instructor “Sandeep Giri”, besides his great experience and knowledge, he was generous, helpful and patient answering all attendees questions in such a way that he could go for more examples and hands-on or even searching the documentation and try new things, I gained much from other attendees’ questions and the way Sandeep responded to them.

This was a great experience having this course and I’m going for more courses in Big Data and Machine Learning with CloudxLab and I recommend it for all my friends and colleagues who look for better learning.

...

Must have for practicing and perfecting hadoop. To setup in PC you need to have a very high end configuration and setup will be pseudo node setup.. For better understanding I recomend CloudxLab

...

They are great. They take care of all the Big Data technologies (Hadoop, Spark, Hive, etc.) so you do not have to worry about installing and running them correclty on your pc. Plus, they have a fantastic customer support. Even when I have had problems debugging my own programs, they have answered me with the correct solution in a few hours, and all of this for a more than reasonable price. I personally recommend it to everyone :)

...

I have been using CloudxLab for last 3 months for learning Hadoop and Spark, and I can vouch for it.

It’s a platform where you can learn from the tutorial videos and then practice in the lab they provide on cloud. The study materials are well-planned and I would be lying if I say its not great.
The video lectures explains the technical stuffs in very simple ways which makes it easier to grasp the concepts. Also, the customer service is great.
So, thumbs up for the team associated with CloudxLab.
To conclude my views, I would just say that, if you are willing to learn Big Data related stuff, I strongly recommend CloudxLab.

...

I think I can give some points on this . Am using cloudxlab for more than an year… my intention is for continuous learning.
For Students and technology change professionals :
In General Big data hadoop, (a) you can learn on your personal PC, but for that the minimum configuration of 12 GB Ram with good processing speed, still when you execute jobs it will take more time for processing jobs as it will be acting as single node.(b) If you try to install each and every components, it will take hell a lot of admin work , and some thing happens , you have to invest lot of time for debugging.
The main advantage of using cloudxlab,
a) Get 6 node production cluster with all installed components, just getting user and password, you can start working on it.
b) You have almost all the access.
c) Good amount of components installed.
d) You can play around with each of them with 5gb of test data.
e) So far I didnt experience any down time.
f) You can Practice in your college lab, on free time.
g) Good email support on technical perspective.
h) They have couple of test data, I use my own.
i) vi and nano editor supported.
j) Some of the components which I remember are HDFS,MapReduce2, YARN, Tez, ZooKeeper,Falcon,Storm, Kafka,Spark,Jupyter Notebook, Hive,HBase, Pig, Sqoop, Oozie, Flume,Accumulo,Ambari.

FAQ

FDP - Program is supported by Ministry of Electronics and Information Technology ( MeitY) in collaboration with CloudxLab, learners will be entitled to get a joint, verifiable certification from IIT Roorkee and CloudxLab after the completion of the FDP program whereas completion of Big Data with Spark Course on CloudxLab will provide you the certificate only from CloudxLab.

Yes the recordings of the sessions will be given to you as a part of your course under "My Courses " page.

Yes, it is mandatory to attend all the live sessions unless there are some uncertain circumstances happening at your end which will be considered by us only when you drop us a mail mentioning your absence for the session. As per the guidelines, you have to secure at least 60% of attendance in order to get a certificate from IIT Roorkee.

You can drop us a mail at discuss.cloudxlab.com mentioning your question, and our technical team will get back to you as soon as possible.

You may apply for the enrollment by submitting the required documents. Documents provided will be verified by concerned authorities, after which we will inform you of the status of your application. In case your documents are not approved by the authorities, you won't be eligible for the certificate recognized by IIT Roorkee.

You can check https://youtu.be/dXCx4anEcgU for watching the Course Preview.

Contact Details

Electronics and ICT Academy

Electronics & ICT Academy Electronics & Communication Department Indian Institute of technology Roorkee 247667, Uttarakhand INDIA

CloudxLab


#215, The Arcade, Brigade Metropolis, Mahadevpura, Bangalore India - 560048 Phone: 080 - 4920 2224

Have more questions? Please contact us at reachus@cloudxlab.com