Common Concepts of HDFS Application Development

Updated on 2024-08-16 GMT+08:00

View PDF

DataNode

A DataNode is used to store data blocks of each file and periodically report the DataNode data storage status to the NameNode.

NameNode

A NameNode is used to manage the namespace, directory structure, and metadata information of a file system and provide the backup mechanism.

Active NameNode: An active NameNode manages the namespace, directory structure, and metadata of file systems, and records the mapping relationships between data blocks and files to which the data blocks belong.
Standby NameNode: A standby NameNode synchronizes data with the active NameNode and takes over services from the active NameNode if the active NameNode becomes abnormal.

JournalNode

A JournalNode synchronizes metadata between the active and standby NameNodes in the High Availability (HA) cluster.

ZKFC

ZKFC must be deployed for each NameNode. It is responsible for monitoring NameNode status and writes status information to ZooKeeper. ZKFC also has permission to select an active NameNode.

Colocation

Colocation is to store associated data or data on which associated operations are performed on the same storage node. The HDFS Colocation stores files to be associated on a same data node so that data can be obtained from the same data node during associated operations. This greatly reduces network bandwidth consumption.

Client

HDFS clients include Java API, C API, shell, HTTP REST API, and web UI.

Java API
Provides application APIs for HDFS. You can follow instructions in HDFS Java APIs to use Java APIs to develop HDFS applications.
C API
Provides application APIs for HDFS. You can follow instructions in HDFS C APIs to use C language to develop applications.
Shell
Provides shell commands to perform operations on HDFS. For details, see HDFS Shell Commands.
HTTP REST API
Provides APIs except shell, Java APIs, and C APIs to monitor HDFS status. For details, see HDFS HTTP REST APIs.
Web UI
Provides a visualized management web page.

Keytab file

The keytab file is a key file that stores user information. Applications use the keytab file to perform API authentication on the MRS Hadoop component.

Parent topic: HDFS Application Development Overview

Previous topic: Introduction to HDFS Application Development

Next topic: HDFS Application Development Process

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot