Updated on 2026-07-29 GMT+08:00

Manually Deploying Hadoop (On Multiple Linux ECSs)

Overview

This best practice guides you through the manual deployment of Hadoop on multiple Linux ECSs. Apache Hadoop is a Java-based, distributed open source software framework. It allows users to develop distributed applications using high-speed computation and storage provided by clusters without knowing the underlying details of the distributed system. The core components of Hadoop include Hadoop Distributed File System (HDFS) and MapReduce.

  • HDFS: a distributed file system that can store and read application data in a distributed way.
  • MapReduce: a distributed computing framework. It allocates computing tasks to servers in a cluster, splits the computing tasks (into Map and Reduce) and performs distributed computing based on JobTracker.

For more information, visit the Apache Hadoop official website.

Prerequisites

  • You have purchased three ECSs and bound EIPs to them.
  • Before deploying Hadoop, you need to plan the nodes. In this example, three ECSs are used. The hadoop001 node is the master node, and the hadoop002 and hadoop003 nodes are worker nodes.

    Functional Component

    hadoop001

    hadoop002

    hadoop003

    HDFS

    • NameNode
    • DataNode

    DataNode

    • SecondaryNameNode
    • DataNode

    YARN

    NodeManager

    • ResourceManager
    • NodeManager

    NodeManager

  • The rules listed in the following table have been added to the security groups that the target ECS belongs to. For details, see Adding a Security Group Rule.
    Table 1 Security group rules

    Direction

    Priority

    Action

    Type

    Protocol & Port

    Source

    Inbound

    1

    Allow

    IPv4

    TCP: 22

    0.0.0.0/0

    Inbound

    1

    Allow

    IPv4

    TCP: 443

    0.0.0.0/0

    Inbound

    1

    Allow

    IPv4

    TCP: 8088

    0.0.0.0/0

    Inbound

    1

    Allow

    IPv4

    TCP: 9870

    0.0.0.0/0

Process

The process of deploying Hadoop on Linux ECSs is as follows:

  1. Install JDK.
  2. Configure passwordless SSH login.
  3. Install Hadoop.
  4. Configure Hadoop.
  5. Start Hadoop.

Procedure

  1. Install JDK.

    Perform the following operations on each ECS.

    1. Log in to an ECS.

      For details, see Login Overview (Linux).

    2. Run the following command to download the JDK 1.8 installation package:
      wget https://download.java.net/openjdk/jdk8u41/ri/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
    3. Run the following command to decompress the downloaded JDK 1.8 installation package:
      tar -zxvf openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
    4. Run the following command to move and rename the JDK installation package.

      In this example, the JDK installation package is renamed java8. You can use other names as required.

      sudo mv java-se-8u41-ri/ /usr/java8
    5. Run the following commands to configure Java environment variables.

      If your JDK installation package is renamed something other than java8, replace java8 with the actual name.

      sudo sh -c "echo 'export JAVA_HOME=/usr/java8' >> /etc/profile"
      sudo sh -c 'echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile'
      source /etc/profile
    6. Run the following command to verify the installation:
      java -version

      The JDK package has been installed if the following information is displayed.

  2. Configure passwordless SSH login.

    Perform the following operations on each ECS.

    1. Run the following command to configure the hostname and communication:
      sudo vim /etc/hosts

      Add <Private-IP-address hostname> of each ECS to the /etc/hosts file. For example:

      Private-IP-address hadoop001
      Private-IP-address hadoop002
      Private-IP-address hadoop003
    2. Run the following command to create a public key and a private key:
      ssh-keygen -t rsa

      Press Enter for three times. If the following information is displayed, the public and private keys have been created.

    3. Run the following command:
      ssh-copy-id <hostname>

      For example:

      Run the following commands on hadoop001 in sequence. After each command is executed, enter yes and the password for logging in to the corresponding ECS.

      ssh-copy-id hadoop001 
      ssh-copy-id hadoop002 
      ssh-copy-id hadoop003

      If the following information is displayed, the passwordless login has been successfully configured.

  3. Install Hadoop.

    Perform the following operations on each ECS.

    1. Run the following command to download the Hadoop package.

      Version 3.2.4 is used as an example.

      wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz
    2. Run the following commands to decompress the Hadoop package to the /opt directory:
      sudo tar -zxvf hadoop-3.2.4.tar.gz -C /opt/
      sudo mv /opt/hadoop-3.2.4 /opt/hadoop
    3. Run the following commands to configure environment variables:
      sudo sh -c "echo 'export HADOOP_HOME=/opt/hadoop' >> /etc/profile"
      sudo sh -c "echo 'export PATH=\$PATH:/opt/hadoop/bin' >> /etc/profile"
      sudo sh -c "echo 'export PATH=\$PATH:/opt/hadoop/sbin' >> /etc/profile"
      source /etc/profile
    4. Run the following commands to change the JAVA_HOME path in the yarn-env.sh and hadoop-env.sh configuration files:
      sudo sh -c 'echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/yarn-env.sh'
      sudo sh -c 'echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/hadoop-env.sh'
    5. Run the following command to verify the installation:
      hadoop version

      Hadoop has been installed if the following information is displayed.

  4. Configure Hadoop.

    Perform the following operations on each ECS.

    1. Modify the Hadoop configuration file core-site.xml.
      1. Run the following command to enter the editing page:
        vim /opt/hadoop/etc/hadoop/core-site.xml
      2. Press i to enter insert mode.
      3. Add the following content between <configuration> and </configuration> in the configuration file:
           <!--NameNode address-->
           <property>
                 <name>fs.defaultFS</name>
                 <value>hdfs://hadoop001:8020</value>
           </property>
           <!--Directory for storing the files generated when Hadoop is used-->
           <property>
                 <name>hadoop.tmp.dir</name>
                 <value>/opt/hadoop/data</value>
           </property>
           <!--Static user for logging in to the HDFS web page: hadoop-->
           <property>
                  <name>hadoop.http.staticuser.user</name>
                  <value>hadoop</value>
           </property>
      4. Press Esc to exit insert mode.
      5. Run the following command to save the settings and exit:
        :wq
    2. Modify the Hadoop configuration file hdfs-site.xml.
      1. Run the following command to enter the editing page:
        vim /opt/hadoop/etc/hadoop/hdfs-site.xml
      2. Press i to enter insert mode.
      3. Add the following content between <configuration> and </configuration> in the configuration file:
            <!-- NameNode web access address-->
            <property>
                <name>dfs.namenode.http-address</name>  
                <value>hadoop001:9870</value>
            </property>
            <!-- Secondary NameNode web access address-->
            <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>hadoop003:9868</value>
            </property>
      4. Press Esc to exit insert mode.
      5. Run the following command to save the settings and exit:
        :wq
    3. Modify the Hadoop configuration file yarn-site.xml.
      1. Run the following command to enter the editing page:
        vim /opt/hadoop/etc/hadoop/yarn-site.xml
      2. Press i to enter insert mode.
      3. Add the following content between <configuration> and </configuration> in the configuration file:
             <!--The NodeManager obtains data in shuffle mode.-->
             <property>
        	 <name>yarn.nodemanager.aux-services</name>
        	 <value>mapreduce_shuffle</value>
             </property>
        
        
            <!--Address of Yarn(ResourceManager)-->
            <property>
        	 <name>yarn.resourcemanager.hostname</name>
        	 <value>hadoop002</value>
            </property> 
        
            <!--Whitelist of environment variables that allowed by NodeManager to pass to containers-->
            <property>
        	<name>yarn.nodemanager.env-whitelist</name>
        	<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
            </property>
      4. Press Esc to exit insert mode.
      5. Run the following command to save the settings and exit:
        :wq
    4. Modify the Hadoop configuration file mapred-site.xml.
      1. Run the following command to enter the editing page:
        vim /opt/hadoop/etc/hadoop/mapred-site.xml
      2. Press i to enter insert mode.
      3. Add the following content between <configuration> and </configuration> in the configuration file:
           <!--Instruct Hadoop to run MR(Map/Reduce) on YARN-->
           <property>
        	 <name>mapreduce.framework.name</name>
        	 <value>yarn</value>
           </property>
      4. Press Esc to exit insert mode.
      5. Run the following command to save the settings and exit:
        :wq
    5. Modify the Hadoop configuration file workers.
      1. Run the following command to enter the editing page:
        vim /opt/hadoop/etc/hadoop/workers
      2. Press i to enter insert mode.
      3. Insert instance information into the workers file.
        hadoop001
        hadoop002
        hadoop003
      4. Press Esc to exit insert mode.
      5. Run the following command to save the settings and exit:
        :wq

  5. Start Hadoop.

    1. Initialize the NameNode.
      The NameNode needs to be initialized only when it is started for the first time. Perform this operation on all ECSs.
      hadoop namenode -format
    2. Start Hadoop.
      • For system security and stability, do not start Hadoop as user root. Attempting to do so may fail due to permission issues. You can start Hadoop as a non-root user, for example, ecs-user.
      • If you need to start Hadoop as user root, you must understand the Hadoop permission control and related risks first and modify the configuration file. For details, see Related Operations.
      • Starting Hadoop as user root poses serious security risks, including but not limited to data breaches, malware gaining system-wide privileges more easily, and unexpected permission issues.
      1. Run start-dfs.sh on hadoop001 to start the HDFS service.

        This script starts the NameNode, SecondaryNameNode, and DataNode components to start the HDFS service.

        start-dfs.sh

        Information similar to the following is displayed.

      2. Run start-yarn.sh on hadoop002 to start the YARN service.

        This script starts the ResourceManager and NodeManager components to start the YARN service.

        start-yarn.sh

        Information similar to the following is displayed.

    3. Run the following command on the three nodes to view the processes that have been successfully started:
      jps

      The processes that have been started are as follows.

    4. Open a browser and enter http://<hadoop002 ECS EIP>:8088 to access the YARN web UI.

      On this web UI, you can view the resource usage, application status (such as MapReduce jobs), and queue information of the entire cluster.

      If information similar to the following is displayed, the Hadoop environment has been deployed.

    5. Open a browser and enter http://<hadoop001 ECS EIP>:9870 to access the NameNode web UI, and enter http://<hadoop003 ECS EIP>:9868 to access the SecondaryNameNode web UI.

      The UI provides information about the HDFS status, cluster health status, active node list, and logs.

      If information similar to the following is displayed, the Hadoop environment has been deployed.

Related Operations

If you need to start Hadoop as user root, do as follows to modify the configuration file:

  1. Run the following commands to edit the start-dfs.sh and stop-dfs.sh files:
    vim /opt/hadoop/sbin/start-dfs.sh
    vim /opt/hadoop/sbin/stop-dfs.sh
  2. Add the following content to the files:
    HDFS_DATANODE_USER=root
    HADOOP_SECURE_DN_USER=hdfs
    HDFS_NAMENODE_USER=root
    HDFS_SECONDARYNAMENODE_USER=root

  3. Run the following commands to edit the start-yarn.sh and stop-yarn.sh files:
    vim /opt/hadoop/sbin/start-yarn.sh
    vim /opt/hadoop/sbin/stop-yarn.sh
  4. Add the following content to the files:
    YARN_RESOURCEMANAGER_USER=root
    HADOOP_SECURE_DN_USER=yarn
    YARN_NODEMANAGER_USER=root