Compiling and Running a MapReduce Application

Updated on 2024-12-10 GMT+08:00

View PDF

Run an application on Linux after application code development is complete.

NOTE:

MapReduce applications can run only on Linux, but not on Windows.

Procedure

Generate an executable MapReduce application package.

Run the mvn package command to generate a JAR file, for example, mapreduce-examples-1.0.jar, and obtain it from the target directory in the project directory.
Upload the generated mapreduce-examples-1.0.jar application package to the Linux client, for example, /opt.
If Kerberos authentication is enabled in the cluster, create a folder (for example, /opt/conf) in the Linux environment to save the user.keytab and krb5.conf files obtained by referring to Preparing a MapReduce Application Development User. Obtain the core-site.xml and hdfs-site.xml files from the client directory in the Linux environment and save them to the preceding folder.
If the sample application specifies OBS as the target file system (for example, obs://<BucketName>/input/) for input and output, you need to configure parameters as follows:

Add AK configuration item fs.obs.access.key and SK configuration item fs.obs.secret.key to $YARN_CONF_DIR/core-site.xml. You can obtain the AK and SK by logging in to the OBS console and go to the My Credentials page.
```
<property>
<name>fs.obs.access.key</name>
<value>xxxxxxxxxxxxxxxx</value>
</property>
<property>
<name>fs.obs.secret.key</name>
<value>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</value>
</property>
```
Execute the sample project on Linux.
- For the MapReduce statistics sample project, run the following command to configure parameters and submit jobs.
  1. If Kerberos authentication is enabled in the cluster, add classpath required for running the sample project in the Linux environment.
    export YARN_USER_CLASSPATH=/opt/conf/
  2. Run the following command:
```
cd /opt
yarn jar mapreduce-examples-1.0.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath>
```
This command is used to set parameters and submit jobs. <inputPath> indicates the input path in HDFS and <outputPath> indicates the output path in HDFS.
NOTE:
- Before running the yarn jar mapreduce-examples-1.0.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath> command, upload the log1.txt and log2.txt files to the <inputPath> directory of HDFS.
- Before running the yarn jar mapreduce-examples-1.0.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath> command, ensure that the <outputPath> directory does not exist. Otherwise, an error will be reported.
- mapreduce-examples-1.0.jar is applicable to MRS 1.x. For MRS 2.x, use mapreduce-examples-2.0.jar.
- Do not restart the HDFS service during the running of MapReduce jobs. Otherwise, the jobs may fail.
- Before running the sample project, you need to modify the authentication information based on site environments.
- For a security cluster with Kerberos authentication enabled, modify principal in the code based on site environments, for example, test@FAA12CC3_0996_432F_9D6F_E18F6F9D7F43.COM.
- For the sample application about multi-component access from MapReduce, perform the following steps.
  1. Obtain the hbase-site.xml, hiveclient.properties, hive-site.xml, and mapred-site.xml files. If the cluster is in security mode, you need to additionally obtain the user.keytab and krb5.conf files and create a folder in the Linux environment to save the configuration file, for example, /opt/conf.
    NOTE:
    
    Contact the administrator to obtain the user.keytab and krb5.conf files of the corresponding account. Obtain the hbase-site.xml file from the HBase client, for example, /opt/client/HBase/hbase/conf, the hiveclient.properties and hive-site.xml files from the Hive client, for example, /opt/client/Hive/config, and the mapred-site.xml file from the Yarn client, for example, /opt/client/Yarn/config.
  2. For a cluster in security mode, create the jaas_mr.conf file in the new folder. The file content is as follows:
```
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="user.keytab"
principal="test@FAA12CC3_0996_432F_9D6F_E18F6F9D7F43.COM"
useTicketCache=false
storeKey=true
debug=true;
}; 
```
    NOTE:
    In the preceding file content, test@HADOOP.COM is an example. Change it based on the site requirements.
    
    Modify principal in the jaas_mr.conf file and code based on the site requirements, for example, test@FAA12CC3_0996_432F_9D6F_E18F6F9D7F43.COM.
    
    Skip this step for a cluster with Kerberos authentication disabled.
  3. In the Linux environment, add the classpath required for running the sample project. For example, if the client installation path is /opt/conf, run the following command:
    export YARN_USER_CLASSPATH=/opt/conf/:/opt/client/HBase/hbase/lib/*:/opt/client/Hive/Beeline/lib/*
    NOTE:
    For an MRS 1.9.x cluster, you need to run the mv /opt/client/Hive/Beeline/lib/derby-10.10.2.0.jar derby-10.10.2.0.jar.bak command before or after running the preceding command.
    
    The JAR package used in the command must be changed based on the actual version in the corresponding path in the cluster.
  4. Submit the MapReduce job and run the following command to run the sample project. Before running the sample project, you need to modify the authentication information based on site environments.
    yarn jar mapreduce-examples-1.0.jar com.huawei.bigdata.mapreduce.examples.MultiComponentExample