Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ HDFS Development Guide/ Preparing an HDFS Application Development Environment/ Preparing an HDFS Application Running Environment
Updated on 2024-08-16 GMT+08:00
View PDF
Share

Preparing an HDFS Application Running Environment

Prerequisites

1. You have installed HDFS on the server and confirmed that HDFS is running properly.

2. You have installed JDK 1.7 or 1.8 on the client operating environment.

3. You have obtained the MRS_Services_Client.tar client installation package.

Scenario

Install the client on Linux.

Procedure

  1. Ensure that the time difference between the client and the Hadoop cluster is less than 5 minutes. You may need to manually modify the client or Hadoop cluster time.

    You can query the MRS cluster time by logging in to the active management node that corresponds to the cluster management IP address to run the date command.

  2. Download the MapReduce client program to the local computer.

    1. Log in to MRS Manager.
    2. Choose Service > Download Client to download the client program to the local PC.

  3. Decompress the MRS_Services_Client.tar client program package. Because the installation package is in .tar format, run the following commands to decompress the package twice:

    tar -xvf MRS_Services_Client.tar

    tar -xvf MRS_Service_ClientConfig.tar

  4. Set environment variables for the operating environment. Assume that the installation package is decompressed in MRS_Services_ClientConfig/.

    Go to the decompressed folder and run the following command to install the client:

    sh install.sh {client_install_home}

  5. Go the client installation directory and run the following command to initialize the environment variables:

    source bigdata_env

  6. Copy the following files from the server to the conf directory in the same directory as the directory of the JAR file. For details about the JAR file exported by the sample project, see Commissioning an HDFS Application on Linux.

    Table 1 Configuration files

    File Name

    Function

    How to Obtain

    core-site.xml

    Configures HDFS parameters.

    ${HADOOP_HOME}/etc/hadoop/core-site.xml

    hdfs-site.xml

    Configures HDFS parameters.

    ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml

    user.keytab

    Provides HDFS user information for Kerberos security authentication.

    If the cluster is in security mode, contact the administrator to obtain the keytab and krb5 files corresponding to the account.

    krb5.conf

    Contains Kerberos server configuration information.
    • In Table 1, ${HADOOP_HOME} indicates the Hadoop installation directory on the server.
    • Keytab authentication is valid for 24 hours. Re-authentication is required 24 hours later.
    • In the sample code, the username of PRNCIPAL_NAME must be the same as the account name of the keytab and krb5 files.
    • The user.keytab and krb5.conf of different clusters cannot be shared.
    • In the sample code, the keytab file used in System.getProperty("user.dir") + File.separator + "conf" + File.separator + "user.keytab" must be the same as the user's keytab file.
    • The log4j.properties file in the conf directory is configured based on the customer requirements.