HDFS Application Development

Updated on 2024-08-10 GMT+08:00

View PDF

Hadoop Distribute File System (HDFS) is a distributed file system that runs on universal hardware. It features high fault tolerance and supports high-throughput data access. It is suitable for processing large-scale data sets.

HDFS is suitable for the following application scenarios:

Processing of massive amounts of data (TB or PB level and larger)
Scenarios that require high throughput
Scenarios that require high reliability
Scenarios that require excellent scalability

MRS provides sample application development projects based on HBase. This practice provides guidance for you to obtain and import a sample project after creating an MRS cluster and then compile and debug the code locally. In this sample project, you can create HDFS directories, and write, read, and delete files.

Creating an MRS Hadoop Cluster

Create and purchase an MRS cluster that contains Hadoop. For details, see Buying a Custom Cluster.
NOTE:

In this practice, an MRS 3.2.0-LTS.1 cluster, with Hadoop installed and with Kerberos authentication enabled, is used as an example.
Click Buy Now and wait until the MRS cluster is created.

Preparing the Application Development Configuration File

Log in to FusionInsight Manager and create a cluster user for security authentication of the sample project.

Choose System > Permission > User. On the displayed page, click Create. On the displayed page, create a machine-machine user, for example, developuser.

Add the hadoop user group to User Group.
Choose System > Permission > User. In the Operation column of developuser, choose More > Download Authentication Credential. Save the file and decompress it to obtain the user.keytab and krb5.conf files of the user.

Choose Cluster. On the Dashboard tab, click More and select Download Client. In the dialog box that is displayed, set Select Client Type to Configuration Files Only and click OK. After the client package is generated, download the package as prompted and decompress it.

For example, if the client configuration file package is FusionInsight_Cluster_1_Services_Client.tar, decompress it to obtain FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles.tar. Then, continue to decompress this file.

Go to the FusionInsight_Cluster_1_Services_ClientConfig_ConfigFiles\HDFS\config directory and obtain the configuration files listed in Table 1.

**Table 1** File
File	Description
core-site.xml	Hadoop Core parameters
hdfs-site.xml	HDFS parameters

Copy all content from the hosts file in the decompression directory to your local hosts file. Ensure that the local PC can communicate with the hosts listed in the hosts file in the decompression directory.
NOTE:
- In this practice, ensure that the local environment can communicate with the network plane where the MRS cluster is deployed. Generally, you can access the MRS cluster via an EIP.
- If the local environment cannot communicate with nodes in the MRS cluster, you can build the sample project first and upload the JAR package to the cluster to run.
- C:\WINDOWS\system32\drivers\etc\hosts is an example directory in a Windows environment for storing the local hosts file.

Obtaining the Sample Project

Obtain the sample project from Huawei Mirrors.

Download the source code and configuration files of the sample project, and configure related development tools on your local PC. For details, see Obtaining Sample Projects from Huawei Mirrors.

Select a branch based on the cluster version and download the required MRS sample project.

For example, the sample project suitable for this practice is hdfs-example-security, which can be obtained at https://github.com/huaweicloud/huaweicloud-mrs-example/tree/mrs-3.2.0.1/src/hdfs-example-security.
Use IDEA to import the sample project and wait for the Maven project to download the dependency packages. For details, see Configuring and Importing Sample Projects.

Figure 1 HDFS sample project

After you configure Maven and SDK parameters on the local PC, the sample project automatically loads related dependency packages.
Place the cluster configuration files and user authentication credentials obtained in Preparing the Application Development Configuration File into the conf directory of the sample project.
Use the required authentication code for the HDFS sample project. Generally, there are security authentication and ZooKeeper authentication.

In this example, you do not need to access HBase or ZooKeeper. Only the basic security authentication code is required.
In the HdfsExample class of the com.huawei.bigdata.hdfs.examples package, change PRNCIPAL_NAME to the username you are using, for example, developuser.
```
private static final String PATH_TO_HDFS_SITE_XML = System.getProperty("user.dir") + File.separator + "conf"
        + File.separator + "hdfs-site.xml";
private static final String PATH_TO_CORE_SITE_XML =  System.getProperty("user.dir") + File.separator + "conf"
        + File.separator + "core-site.xml";
private static final String PRNCIPAL_NAME = "developuser";
private static final String PATH_TO_KEYTAB = System.getProperty("user.dir") + File.separator + "conf"
        + File.separator + "user.keytab";
private static final String PATH_TO_KRB5_CONF = System.getProperty("user.dir") + File.separator + "conf"
        + File.separator + "krb5.conf";
...
```
In this sample project, the development roadmap based on the service requirement is as follows.

The following example describes how to read, write, delete the /user/hdfs-examples/test.txt file in HDFS.
1. Pass the cluster security authentication.
2. Create a FileSystem object: fSystem
3. Call the mkdir API in fSystem to create a directory.
4. Call create on fSystem to create an FSDataOutputStream object out. Write data into out by calling write.
5. Call append on fSystem to create an FSDataOutputStream object out. Append data into out by calling write.
6. Call open on fSystem to create an FSDataInputStream object in. Read files of in by calling read.
7. Call delete on fSystem to delete a file.
8. Call delete on fSystem to delete a folder.

Building and Running the Application

Click Reimport All Maven Projects in the Maven window on the right of IDEA to load the Maven project dependencies.

Figure 2 Loading a sample project
Compile and run the application.
1. Choose Maven, locate the target project name, and double-click clean under Lifecycle to run the clean command of Maven.
2. Choose Maven, locate the target project name, and double-click compile under Lifecycle to run the compile command of Maven.
After the building is complete, message "Build Success" is displayed and the target directory is generated.
```
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21.276 s
[INFO] Finished at: 2023-05-05T14:36:39+08:00
[INFO] ------------------------------------------------------------------------
```
Run the application.

Right-click the HdfsExample.java file and choose Run 'HdfsExample.main()' from the shortcut menu.

Figure 3 Running the application

Check the output information after running the sample. The following information indicates that related file operations are successfully executed:

...
2217 [main] INFO  org.apache.hadoop.security.UserGroupInformation  - Login successful for user developuser using keytab file user.keytab. Keytab auto renewal enabled : false
2217 [main] INFO  com.huawei.hadoop.security.LoginUtil  - Login success!!!!!!!!!!!!!!
3529 [main] WARN  org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory  - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
4632 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples
5392 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
8200 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
9384 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
9384 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
9636 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples\test.txt
9860 [main] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples
10010 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_0
10069 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to create path /user/hdfs-examples/hdfs_example_1
10553 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
10607 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to write.
13356 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
13469 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to append.
13784 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
13784 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
13834 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - result is : hi, I am bigdata. It is successful if you can see me.I append this content.
13834 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to read.
13837 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_0\test.txt
13889 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete the file /user/hdfs-examples/hdfs_example_1\test.txt
14003 [hdfs_example_0] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_0
14118 [hdfs_example_1] INFO  com.huawei.bigdata.hdfs.examples.HdfsExample  - success to delete path /user/hdfs-examples/hdfs_example_1
...

Parent topic: Rapidly Develop MRS Component Applications

Previous topic: HBase Application Development

Next topic: Hive JDBC Application Development