An HDFS cluster has two types of nodes: one namenode also known as the master and multiple datanodes
An HDFS cluster consists of many machines. One of these machines is designated as namenode and other machines act as datanodes. Please note that we can also have datanode on the machine where namenode service is running. By default, namenode metadata service runs on port 8020
The namenode manages the filesystem namespace. It maintains the filesystem tree and the metadata for all the files and directories in the tree. This information is stored in RAM and persisted on the local disk in the form of two files: the namespace image and the edit log. The namenode also knows the datanodes on which all the blocks for a given file are located.
Datanodes are the workhorses of the filesystem. While namenode keeps the index of which block is stored in which datanode, datanodes store the actual data. In short, datanodes do not know the name of a file and namenode does not know what is inside a file.
As a rule of thumb, namenode should be installed on the machine having bigger RAM as all the metadata for files and directories is stored in RAM. We will not be able to store many files in HDFS if RAM is not big as there will not be enough space for metadata to fit in RAM. Since data nodes store actual data, datanodes should be run on the machines having the bigger disk.
Please login to comment
4 Comments
C:\hadoop-3.3.6\sbin>hdfs dfs –mkdir /sample
2023-07-30 08:31:43,523 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
?mkdir: Unknown command
i am facing this issue... pls suggest
Upvote ShareHi Indu,
I cannot see any particular issue with the command related to the error. You can try running the code in our lab terminal. It is working fine here.
Upvote ShareWriting "An" before HDFS, is it right?
Upvote ShareYes, it is. We use “An” before an acronym that begins with “H” because “H” is pronounced with a vowel sound “aitch”.
Upvote Share