Vagdevi K, Author at CloudxLab Blog

Practice questions for Machine Learning Engineer Roles

Welcome!

You might have come across several posts which focus only on the theoretical questions for you to prepare for a machine learning engineer role. But is the theoretical preparation enough?

The ML Engineers in the real world do much more than just making models. They spend enough time understanding the data before actually building a model. For this, they should be able to perform different operations on the data, make intuitions and manipulate the data as per the needs. So an ML Engineer must be able to how to play with data and tell some intuition stories.

Pandas is a library for Python to perform various operations on data. Numpy is a famous Python library for numerical computations. It is often expected that an ML Engineer is well-versed with both of these libraries. But where to practice?

ClouldxLab offers a solution. We have come up with some amazing questions which would help you practice Python, Pandas and Numpy hands-on and make you interview ready.

So what are you waiting for? Encourage the aspiring ML Engineer in you, by waking up the problem solver in you. Practice the following questions: https://cloudxlab.com/assessment/playlist-intro/862/machine-learning-prerequisite-mains-10th-july-2021.

All the best!

Getting Started with various tools at CloudxLab

Welcome!

We are happy to announce that we have come up with a new consolidated playlist, which summaries about various tools present at CloudxLab environment, how to use them and where to learn about them.

This would be incrementally improved as new technologies keep getting installed on the lab.

You may find the playlist here.

In this playlist, there is a dedicated slide for each technology. For example, if you want to understand how to use Pandas on the lab, go to the slide named Pandas.

Upon clicking on Pandas, you would be able to see the Pandas guide as follows:

As you could see, this slide contains all the basic information needed such as:

the purpose of the library
link for the official home page
link for the official documentation
related resources you could use to learn about the library.
instructions on how to use it on the CloudxLab environment.
1-2 lines of sample examples to use it, such as how to inport the library and how to check the version.

We hope that this will be a great starting guide for our users and makes their job of getting started easier.

Happy learning!

How does YARN interact with Zookeeper to support High Availability?

In the Hadoop ecosystem, YARN, short for Yet Another Resource negotiator, holds the responsibility of resource allocation and job scheduling/management. The Resource Manager(RM), one of the components of YARN, is primarily responsible for accomplishing these tasks of coordinating with the various nodes and interacting with the client.

To learn more about YARN, feel free to visit here.

Hence, Resource Manager in YARN is a single point of failure – meaning, if the Resource Manager is down for some reason, the whole of the system gets disturbed due to interruption in the resource allocation or job management, and thus we cannot run any jobs on the cluster.

To avoid this issue, we need to enable the High Availability(HA) feature in YARN. When HA is enabled, we run another Resource Manager parallelly on another node, and this is known as Standby Resource Manager. The idea is that, when the Active Resource Manager is down, the Standby Resource Manager becomes active, and ensures smooth operations on the cluster. And the process continues.

How to design a large-scale system to process emails using multiple machines [Zookeeper Use Case Study]?

Introduction

As part of this blog we are going to discuss various ways of large scale system design and the pros-cons of each.

To get a fair understanding of this post, you should know what is distributed computing, what is deadlock and race conditions, locking in distributed systems and Zookeeper etc. Let’s get started.

Scenario

Consider a situation where we have an email inbox that consists of emails, and emails are to be processed. For example, processing those emails and classifying each of the emails as spam or non-spam. The other example of the processing could be we are indexing the email so that the search could be performed.

We have an email-processor program, running on various machines distributed physically from each other.

Email processor program running on distributed systems

Now these machines need to somehow coordinate such that:

No email is processed two times
No email is left unprocessed

Introduction to Apache Zookeeper

In the Hadoop ecosystem, Apache Zookeeper plays an important role in coordination amongst distributed resources. Apart from being an important component of Hadoop, it is also a very good concept to learn for a system design interview.

If you would prefer the videos with hands-on, feel free to jump in here.

Alright, so let’s get started.

Goals

In this post, we will understand the following:

What is Apache Zookeeper?
How Zookeeper achieves coordination?
Zookeeper Architecture
Zookeeper Data Model
Some Hands-on with Zookeeper
Election & Majority in Zookeeper
Zookeeper Sessions
Application of Zookeeper
What kind of guarantees does ZooKeeper provide?
Operations provided by Zookeeper
Zookeeper APIs
Zookeeper Watches
ACL in Zookeeper
Zookeeper Usecases

Distributed Computing with Locks

Introduction

Having known of the prevalence of BigData in real-world scenarios, it’s time for us to understand how they work. This is a very important topic in understanding the principles behind system design and coordination among machines in big data. So let’s dive in.

Scenario:

Consider a scenario where there is a resource of data, and there is a worker machine that has to accomplish some task using that resource. For example, this worker is to process the data by accessing that resource. Remember that the data source is having huge data; that is, the data to be processed for the task is very huge.

Race Condition and Deadlock

A good system needs to make sure that race condition and deadlock can’t occur. In this post, let us learn about Race Condition and Deadlock.

Understanding Big Data Stack – Apache Hadoop and Spark

Introduction

There are many Big Data Solution stacks.

The first and most powerful stack is Apache Hadoop and Spark together. While Hadoop provides storage for structured and unstructured data, Spark provides the computational capability on top of Hadoop.

Introduction to Big Data and Distributed Systems

Introduction

As everyone knows, Big Data is a term of fascination in the present-day era of computing. It is in high demand in today’s IT industry and is believed to revolutionize technical solutions like never before.

Improving the Performance of Deep-Learning based Flask App with ZMQ

Introduction

It is a well-known fact that deep learning models are heavy; with a lot of weights for the deep layers. And it is obviously an overhead to load the model every time we need to get the predictions from the model. Thus this is costly in terms of the time of execution.

In this project, we will mainly focus on addressing this issue, by uniquely integrating the networking functionalities provided by ZMQ library. We will build a server-client based architecture to make the model load exactly once(that is during the starting of the app). The predictions from the model will be served by the model server, as long as it listens to its Flask client which requests it for the predictions for an input image.