Configuring the Distributed Cache to Execute MapReduce Jobs

Scenarios

This section applies to MRS 3.x or later.

Distributed caching is useful in the following scenarios:

Rolling upgrade
During the upgrade, applications must keep the text content (JAR file or configuration file) unchanged. The content is not based on YARN of the current version, but based on the version when YARN is submitted. Generally, applications (such as MapReduce, Hive, and Tez) need to be installed locally. Libraries need to be installed on all cluster servers (clients and servers). When a rolling upgrade or downgrade starts in the cluster, the version of the locally installed library changes during application running. During the rolling upgrade, only a few NodeManagers are upgraded first. These NodeManagers obtain the software of the latest version. This leads to inconsistent behavior and can result in run-time errors.
Co-existence of multiple YARN versions
Cluster administrators may run tasks that use multiple versions of YARN and Hadoop JARs in a cluster. However, this task is difficult to be implemented because the JARs have been localized and have only one version.

The MapReduce application framework can be deployed through the distributed cache and does not depend on the static version copied during installation. Therefore, you can store multiple versions of Hadoop in HDFS and configure the mapred-site.xml file to specify the default version used by the task. You can run different versions of MapReduce by setting proper configuration attributes without using the versions deployed in the cluster.

Figure 1 Clusters with NodeManagers and Applications of multiple versions

As shown in Figure 1, the application can use Hadoop JARs in HDFS instead of the local version. Therefore, during the rolling upgrade, even if NodeManager has been upgraded, the application can still run Hadoop of the earlier version.

Configuration Description

Log in to the HDFS client.
1. Log in to the node where the client is installed as the client installation user.
2. Go to the client installation directory.
  cd Client installation path
3. Configure environment variables.
  source bigdata_env
4. If the cluster is in security mode, authenticate the user. Skip user authentication for normal clusters.
  kinit Component service user
Save the MapReduce .tar package of the specified version to a directory that can be accessed by applications in HDFS.

$HADOOP_HOME/bin/hdfs dfs -put hadoop-x.tar.gz /mapred/framework/

Set parameters in the Client installation path/Yarn/config/mapred-site.xml file by referring to Table 1.

vi Client installation path/Yarn/config/mapred-site.xml

**Table 1** Distributed cache parameters
Parameter	Description	Default Value
mapreduce.application.framework.path	Indicates the URL directing to the archive location. NOTE: This property can also create an alias for the archive if the URL fragment identity name is specified as follows. In this example, the alias is set to mr-framework. <property> <name>mapreduce.application.framework.path</name> <value>hdfs:/mapred/framework/hadoop-x.tar.gz#mr-framework</value> </property>	N/A
mapreduce.application.classpath	Indicates the parameter property, which contains the MapReduce JARs in the class directory. NOTE: For example, the alias mr-framework used in the framework path is used to match the directory. <property> <name>mapreduce.application.classpath</name> <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/:$PWD/mr-framework/hadoop/share/hadoop/common/:$PWD/mr-framework/hadoop/share/hadoop/common/lib/:$PWD/mr-framework/hadoop/share/hadoop/yarn/:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/:/etc/hadoop/conf/secure</value></property>	N/A

Table 1 Distributed cache parameters

Parameter

Description

Default Value

mapreduce.application.framework.path

Indicates the URL directing to the archive location.

NOTE:

This property can also create an alias for the archive if the URL fragment identity name is specified as follows. In this example, the alias is set to mr-framework.

<property> <name>mapreduce.application.framework.path</name> <value>hdfs:/mapred/framework/hadoop-x.tar.gz#mr-framework</value> </property>

N/A

mapreduce.application.classpath

Indicates the parameter property, which contains the MapReduce JARs in the class directory.

NOTE:

For example, the alias mr-framework used in the framework path is used to match the directory.

<property> <name>mapreduce.application.classpath</name>   <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/etc/hadoop/conf/secure</value></property>

N/A

You can upload MapReduce .tar packages of multiple versions to HDFS. Different mapred-site.xml files indicate different locations. After that, you can run tasks for a specific mapred-site.xml file. The following is an example of running MapReduce tasks using the MapReduce .tar package of the x version:

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi -conf etc/hadoop-x/mapred-site.xml 10 10