Manually Deploying Hadoop (On Multiple Linux ECSs)
Overview
This best practice guides you through the manual deployment of Hadoop on multiple Linux ECSs. Apache Hadoop is a Java-based, distributed open source software framework. It allows users to develop distributed applications using high-speed computation and storage provided by clusters without knowing the underlying details of the distributed system. The core components of Hadoop include Hadoop Distributed File System (HDFS) and MapReduce.
- HDFS: a distributed file system that can store and read application data in a distributed way.
- MapReduce: a distributed computing framework. It allocates computing tasks to servers in a cluster, splits the computing tasks (into Map and Reduce) and performs distributed computing based on JobTracker.
For more information, visit the Apache Hadoop official website.
Prerequisites
- You have purchased three ECSs and bound EIPs to them.
- Before deploying Hadoop, you need to plan the nodes. In this example, three ECSs are used. The hadoop001 node is the master node, and the hadoop002 and hadoop003 nodes are worker nodes.
Functional Component
hadoop001
hadoop002
hadoop003
HDFS
- NameNode
- DataNode
DataNode
- SecondaryNameNode
- DataNode
YARN
NodeManager
- ResourceManager
- NodeManager
NodeManager
- The rules listed in the following table have been added to the security groups that the target ECS belongs to. For details, see Adding a Security Group Rule.
Table 1 Security group rules Direction
Priority
Action
Type
Protocol & Port
Source
Inbound
1
Allow
IPv4
TCP: 22
0.0.0.0/0
Inbound
1
Allow
IPv4
TCP: 443
0.0.0.0/0
Inbound
1
Allow
IPv4
TCP: 8088
0.0.0.0/0
Inbound
1
Allow
IPv4
TCP: 9870
0.0.0.0/0
Process
The process of deploying Hadoop on Linux ECSs is as follows:
Procedure
- Install JDK.
Perform the following operations on each ECS.
- Log in to an ECS.
For details, see Login Overview (Linux).
- Run the following command to download the JDK 1.8 installation package:
wget https://download.java.net/openjdk/jdk8u41/ri/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
- Run the following command to decompress the downloaded JDK 1.8 installation package:
tar -zxvf openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
- Run the following command to move and rename the JDK installation package.
In this example, the JDK installation package is renamed java8. You can use other names as required.
sudo mv java-se-8u41-ri/ /usr/java8
- Run the following commands to configure Java environment variables.
If your JDK installation package is renamed something other than java8, replace java8 with the actual name.
sudo sh -c "echo 'export JAVA_HOME=/usr/java8' >> /etc/profile" sudo sh -c 'echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile' source /etc/profile
- Run the following command to verify the installation:
java -version
The JDK package has been installed if the following information is displayed.

- Log in to an ECS.
- Configure passwordless SSH login.
Perform the following operations on each ECS.
- Run the following command to configure the hostname and communication:
sudo vim /etc/hosts
Add <Private-IP-address hostname> of each ECS to the /etc/hosts file. For example:
Private-IP-address hadoop001 Private-IP-address hadoop002 Private-IP-address hadoop003
- Run the following command to create a public key and a private key:
ssh-keygen -t rsa
Press Enter for three times. If the following information is displayed, the public and private keys have been created.

- Run the following command:
ssh-copy-id <hostname>
For example:
Run the following commands on hadoop001 in sequence. After each command is executed, enter yes and the password for logging in to the corresponding ECS.
ssh-copy-id hadoop001 ssh-copy-id hadoop002 ssh-copy-id hadoop003
If the following information is displayed, the passwordless login has been successfully configured.

- Run the following command to configure the hostname and communication:
- Install Hadoop.
Perform the following operations on each ECS.
- Run the following command to download the Hadoop package.
Version 3.2.4 is used as an example.
wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz
- Run the following commands to decompress the Hadoop package to the /opt directory:
sudo tar -zxvf hadoop-3.2.4.tar.gz -C /opt/ sudo mv /opt/hadoop-3.2.4 /opt/hadoop
- Run the following commands to configure environment variables:
sudo sh -c "echo 'export HADOOP_HOME=/opt/hadoop' >> /etc/profile" sudo sh -c "echo 'export PATH=\$PATH:/opt/hadoop/bin' >> /etc/profile" sudo sh -c "echo 'export PATH=\$PATH:/opt/hadoop/sbin' >> /etc/profile" source /etc/profile
- Run the following commands to change the JAVA_HOME path in the yarn-env.sh and hadoop-env.sh configuration files:
sudo sh -c 'echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/yarn-env.sh' sudo sh -c 'echo "export JAVA_HOME=/usr/java8" >> /opt/hadoop/etc/hadoop/hadoop-env.sh'
- Run the following command to verify the installation:
hadoop version
Hadoop has been installed if the following information is displayed.

- Run the following command to download the Hadoop package.
- Configure Hadoop.
Perform the following operations on each ECS.
- Modify the Hadoop configuration file core-site.xml.
- Run the following command to enter the editing page:
vim /opt/hadoop/etc/hadoop/core-site.xml
- Press i to enter insert mode.
- Add the following content between <configuration> and </configuration> in the configuration file:
<!--NameNode address--> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop001:8020</value> </property> <!--Directory for storing the files generated when Hadoop is used--> <property> <name>hadoop.tmp.dir</name> <value>/opt/hadoop/data</value> </property> <!--Static user for logging in to the HDFS web page: hadoop--> <property> <name>hadoop.http.staticuser.user</name> <value>hadoop</value> </property> - Press Esc to exit insert mode.
- Run the following command to save the settings and exit:
:wq
- Run the following command to enter the editing page:
- Modify the Hadoop configuration file hdfs-site.xml.
- Run the following command to enter the editing page:
vim /opt/hadoop/etc/hadoop/hdfs-site.xml
- Press i to enter insert mode.
- Add the following content between <configuration> and </configuration> in the configuration file:
<!-- NameNode web access address--> <property> <name>dfs.namenode.http-address</name> <value>hadoop001:9870</value> </property> <!-- Secondary NameNode web access address--> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop003:9868</value> </property> - Press Esc to exit insert mode.
- Run the following command to save the settings and exit:
:wq
- Run the following command to enter the editing page:
- Modify the Hadoop configuration file yarn-site.xml.
- Run the following command to enter the editing page:
vim /opt/hadoop/etc/hadoop/yarn-site.xml
- Press i to enter insert mode.
- Add the following content between <configuration> and </configuration> in the configuration file:
<!--The NodeManager obtains data in shuffle mode.--> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!--Address of Yarn(ResourceManager)--> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop002</value> </property> <!--Whitelist of environment variables that allowed by NodeManager to pass to containers--> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> - Press Esc to exit insert mode.
- Run the following command to save the settings and exit:
:wq
- Run the following command to enter the editing page:
- Modify the Hadoop configuration file mapred-site.xml.
- Run the following command to enter the editing page:
vim /opt/hadoop/etc/hadoop/mapred-site.xml
- Press i to enter insert mode.
- Add the following content between <configuration> and </configuration> in the configuration file:
<!--Instruct Hadoop to run MR(Map/Reduce) on YARN--> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
- Press Esc to exit insert mode.
- Run the following command to save the settings and exit:
:wq
- Run the following command to enter the editing page:
- Modify the Hadoop configuration file workers.
- Run the following command to enter the editing page:
vim /opt/hadoop/etc/hadoop/workers
- Press i to enter insert mode.
- Insert instance information into the workers file.
hadoop001 hadoop002 hadoop003
- Press Esc to exit insert mode.
- Run the following command to save the settings and exit:
:wq
- Run the following command to enter the editing page:
- Modify the Hadoop configuration file core-site.xml.
- Start Hadoop.
- Initialize the NameNode.
The NameNode needs to be initialized only when it is started for the first time. Perform this operation on all ECSs.hadoop namenode -format
- Start Hadoop.
- For system security and stability, do not start Hadoop as user root. Attempting to do so may fail due to permission issues. You can start Hadoop as a non-root user, for example, ecs-user.
- If you need to start Hadoop as user root, you must understand the Hadoop permission control and related risks first and modify the configuration file. For details, see Related Operations.
- Starting Hadoop as user root poses serious security risks, including but not limited to data breaches, malware gaining system-wide privileges more easily, and unexpected permission issues.
- Run start-dfs.sh on hadoop001 to start the HDFS service.
This script starts the NameNode, SecondaryNameNode, and DataNode components to start the HDFS service.
start-dfs.sh
Information similar to the following is displayed.

- Run start-yarn.sh on hadoop002 to start the YARN service.
This script starts the ResourceManager and NodeManager components to start the YARN service.
start-yarn.sh
Information similar to the following is displayed.

- Run the following command on the three nodes to view the processes that have been successfully started:
jps
The processes that have been started are as follows.

- Open a browser and enter http://<hadoop002 ECS EIP>:8088 to access the YARN web UI.
On this web UI, you can view the resource usage, application status (such as MapReduce jobs), and queue information of the entire cluster.
If information similar to the following is displayed, the Hadoop environment has been deployed.

- Open a browser and enter http://<hadoop001 ECS EIP>:9870 to access the NameNode web UI, and enter http://<hadoop003 ECS EIP>:9868 to access the SecondaryNameNode web UI.
The UI provides information about the HDFS status, cluster health status, active node list, and logs.
If information similar to the following is displayed, the Hadoop environment has been deployed.

- Initialize the NameNode.
Related Operations
If you need to start Hadoop as user root, do as follows to modify the configuration file:
- Run the following commands to edit the start-dfs.sh and stop-dfs.sh files:
vim /opt/hadoop/sbin/start-dfs.sh vim /opt/hadoop/sbin/stop-dfs.sh
- Add the following content to the files:
HDFS_DATANODE_USER=root HADOOP_SECURE_DN_USER=hdfs HDFS_NAMENODE_USER=root HDFS_SECONDARYNAMENODE_USER=root

- Run the following commands to edit the start-yarn.sh and stop-yarn.sh files:
vim /opt/hadoop/sbin/start-yarn.sh vim /opt/hadoop/sbin/stop-yarn.sh
- Add the following content to the files:
YARN_RESOURCEMANAGER_USER=root HADOOP_SECURE_DN_USER=yarn YARN_NODEMANAGER_USER=root
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot