YARN - Architecture
Let us try to understand the architecture of YARN. The objective of YARN is to be able to execute any kind of workfload in distributed fashion.
All of the components that you see in the diagram are software programs. It has one resource manager. Each machine has a node manager. The node managers create containers to execute the programs. Application Masters are the programs per application that are executed inside the containers. Examples of Application Masters are Mapreduce AM and Spark AM.
The flow goes like this: A client submits an application to the resource manager. The resource manager consults with the node managers and creates an Application Master for the client inside a container.
Now, the application master registers itself with resource manager so that the client can monitor the progress and interact with the application master.
Application Master then requests Resource Manager for containers. The resource manager consults and negotiates with Node Managers for the containers. Node Managers create containers and launch app in these containers. The node managers talk back to the resource manager about the resource usage. To check the status, the clients can talk to Application Master. On completion, Application Master deregisters itself from the resource manager and shuts down.
It can be better understood if we think of YARN as a country where Resource Manager is the president. And clients are organizations like Microsoft or Shell who want their work to be done by the way of establishing offices. Each node is a state such as California and the node manager is governor of the state. The container is the place for the office. The application master is the head office.
Though it looks like over simplification, it would make us understand how YARN works. Microsoft wants to get their offices to be setup, so they contact the president, the president talks to governors of the states and then the head office is established. After wards to get more offices established in various states, head office would request president and the president would discuss with governors and offices would be established.
Loading comments...