Preparing an HDFS Application Running Environment

Updated on 2024-12-10 GMT+08:00

View PDF

Prerequisites

1. You have installed HDFS on the server and confirmed that HDFS is running properly.

2. You have installed JDK 1.7 or 1.8 on the client operating environment.

3. You have obtained the MRS_Services_Client.tar client installation package.

Scenario

Install the client on Linux.

Procedure

Ensure that the time difference between the client and the Hadoop cluster is less than 5 minutes. You may need to manually modify the client or Hadoop cluster time.

You can query the MRS cluster time by logging in to the active management node that corresponds to the cluster management IP address to run the date command.
Download the MapReduce client program to the local computer.
1. Log in to MRS Manager.
2. Choose Service > Download Client to download the client program to the local PC.
Decompress the MRS_Services_Client.tar client program package. Because the installation package is in .tar format, run the following commands to decompress the package twice:

tar -xvf MRS_Services_Client.tar

tar -xvf MRS_Service_ClientConfig.tar
Set environment variables for the operating environment. Assume that the installation package is decompressed in MRS_Services_ClientConfig/.

Go to the decompressed folder and run the following command to install the client:

sh install.sh {client_install_home}
Go the client installation directory and run the following command to initialize the environment variables:

source bigdata_env

Copy the following files from the server to the conf directory in the same directory as the directory of the JAR file. For details about the JAR file exported by the sample project, see Commissioning an HDFS Application on Linux.

**Table 1** Configuration files
File Name	Function	How to Obtain
core-site.xml	Configures HDFS parameters.	${HADOOP_HOME}/etc/hadoop/core-site.xml
hdfs-site.xml	Configures HDFS parameters.	${HADOOP_HOME}/etc/hadoop/hdfs-site.xml
user.keytab	Provides HDFS user information for Kerberos security authentication.	If the cluster is in security mode, contact the administrator to obtain the keytab and krb5 files corresponding to the account.
krb5.conf	Contains Kerberos server configuration information.

NOTE:

In Table 1, ${HADOOP_HOME} indicates the Hadoop installation directory on the server.

Keytab authentication is valid for 24 hours. Re-authentication is required 24 hours later.
In the sample code, the username of PRNCIPAL_NAME must be the same as the account name of the keytab and krb5 files.
The user.keytab and krb5.conf of different clusters cannot be shared.
In the sample code, the keytab file used in System.getProperty("user.dir") + File.separator + "conf" + File.separator + "user.keytab" must be the same as the user's keytab file.
The log4j.properties file in the conf directory can be configured based on the customer requirements.

Parent topic: Preparing an HDFS Application Development Environment

Previous topic: Preparing the Eclipse and JDK

Next topic: Importing and Configuring HDFS Sample Projects

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot