1) I don't exactly get what you mean by file read and write operations. Zookeeper is basically a central distributed service used to achieve synchronization among the nodes of a cluster.
2) Yes, Yarn does interact with Zookeeper. YARN handles resource allocation and scheduling for a cluster of nodes. The resource manager of yarn is a single point of failure: if the resource manager is down, the whole system would be disturbed as the jobs/resource allocation could no longer be scheduled/managed. So, the resource manager has a standby resource manager which could become active if the existing active resource manager is no longer up. But there should be no split brain scenario: ie, there should be no situation where the cluster is managed by 2 resource managers(the active one and standby one). At any point of time, there should be one and only one resource manager. And, the state of the standby resource manager should be up to date with the active one. So there is a need for a central place where these configuration details could be stored. Zookeeper helps achieve this. The active resource manager acquires an ephemeral znode lock with zookeeper. The standby RM places a watch on this ephemeral znode. As long as this lock is there, no other standby resource manager could acquire the lock. If the active resource manager turned down, the session of this ephemeral znode would no longer be active and thus zookeeper deletes this znode. As soon as this ephemeral znode created by former RM is deleted, the standby RM would be notified about this action, and the standby RM now creates an ephemeral znode, and then the standby resource manager could turn active. Feel free to visit: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
3) Znode stores configuration data like version, status information, configuration, location information, etc., so the data stored at each node is usually small, in the byte to kilobyte range.
It's always good to learn Java as it has application not only in Zookeeper but in other related areas too. We do have an optional Java Essentials playlist, you may want to complete that.
Pardon me if my earlier question fails to put my point, Sir I had completed java essentials module, I want to ask is zookeeper an inbuilt tool itself or we have to invoke zookeeper feature through Java coding.
It depends on how you are using Zookeeper. If you are using it as a standalone application, you would not need to use Java but will be able to access Zookeeper through CLI. However, if you want to create client applications for Zookeeper, you might want to take a look at Zookeeper Java API.
Please login to comment
9 Comments
Hi,
1) Would you please help me understand the role zookeeper plays in 1. file read 2. file write operations?
2) Does Yarn interact with the zookeeper? If so, how?
3) What kind of data the Znode stores? Can you explain with an example?
Thanks
Upvote ShareThis comment has been removed.
Hi,
1) I don't exactly get what you mean by file read and write operations. Zookeeper is basically a central distributed service used to achieve synchronization among the nodes of a cluster.
2) Yes, Yarn does interact with Zookeeper. YARN handles resource allocation and scheduling for a cluster of nodes. The resource manager of yarn is a single point of failure: if the resource manager is down, the whole system would be disturbed as the jobs/resource allocation could no longer be scheduled/managed. So, the resource manager has a standby resource manager which could become active if the existing active resource manager is no longer up. But there should be no split brain scenario: ie, there should be no situation where the cluster is managed by 2 resource managers(the active one and standby one). At any point of time, there should be one and only one resource manager. And, the state of the standby resource manager should be up to date with the active one. So there is a need for a central place where these configuration details could be stored. Zookeeper helps achieve this. The active resource manager acquires an ephemeral znode lock with zookeeper. The standby RM places a watch on this ephemeral znode. As long as this lock is there, no other standby resource manager could acquire the lock. If the active resource manager turned down, the session of this ephemeral znode would no longer be active and thus zookeeper deletes this znode. As soon as this ephemeral znode created by former RM is deleted, the standby RM would be notified about this action, and the standby RM now creates an ephemeral znode, and then the standby resource manager could turn active. Feel free to visit: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
3) Znode stores configuration data like version, status information, configuration, location information, etc., so the data stored at each node is usually small, in the byte to kilobyte range.
Thanks.
1 Upvote ShareSir/Mam do we have to also learn zookeeper's Java Coding as mentioned in O'Reilly zookeeper.
Upvote ShareHi,
It's always good to learn Java as it has application not only in Zookeeper but in other related areas too. We do have an optional Java Essentials playlist, you may want to complete that.
Thanks.
Upvote SharePardon me if my earlier question fails to put my point, Sir I had completed java essentials module, I want to ask is zookeeper an inbuilt tool itself or we have to invoke zookeeper feature through Java coding.
Upvote ShareHi,
It depends on how you are using Zookeeper. If you are using it as a standalone application, you would not need to use Java but will be able to access Zookeeper through CLI. However, if you want to create client applications for Zookeeper, you might want to take a look at Zookeeper Java API.
Thanks.
2 Upvote Shareshowing only 98% complete. request to rectify
Upvote ShareHi,
You have not completed topic 15.
Thanks.
Upvote Share