Compiling and Running the Application

Scenario

After the code development is complete, you can run an application in the Linux development environment.

Prerequisites

You have installed the Yarn client.

Procedure

1. Go to the local root directory of the project and run the following command in Windows cmd to compress the package:

   mvn -s "{maven_setting_path}" clean package

   

   • In the preceding command, {maven_setting_path} is the path of the setting.xml file of the local maven.
   • After the package is successfully packed, obtain the JAR package, for example, MRTest-XXX.jar, from the target subdirectory in the root directory of the project. The name of the JAR package varies according to the actual package.

2. Upload MRTest-XXX.jar to the Linux client, such as /opt/client/conf, the same directory where the configuration files are located in.
3. Run the sample project in the Linux environment.
   • Run the following command for MapReduce statistics sample project to configure parameters and submit jobs:

     yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath>

     <inputPath> indicates the input path and <outputPath> indicates the output path in HDFS.

     

     • Before running the yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath> command, upload the log1.txt and log2.txt files to the <inputPath> directory in HDFS.
     • Before running the yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath> command, ensure that the <outputPath> directory is deleted. Otherwise, an error will occur.
     • Do not restart the HDFS service during the running of MapReduce tasks. Otherwise, the tasks may fail.
   • In MapReduce accessing multi-components sample project, perform the following operations:
     1. Obtain the user.keytab, krb5.conf, hbase-site.xml, hiveclient.properties, and hive-site.xml configuration files, create a folder for example /opt/client/conf, and save the configurations files.
        

        The account permission files user.keytab and krb5.conf are acquired from the administrator. The account permission file hbase-site.xml is acquired from the HBase client, hiveclient.properties and hive-site.xml are acquired from the Hive client.

     2. Create the jaas_mr.conf file in the created folder. The content of the jaas_mr.conf file is as follows:

        Client {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        keyTab="user.keytab"
        principal="test@<system domain name>"
        useTicketCache=false
        storeKey=true
        debug=true;
        };

        

        The test@<system domain name> in the preceding file content is an example of user. Modify it as required.

     3. Add the classpath required for sample projects in the Linux environment, Example of classpath:

        export YARN_USER_CLASSPATH=/opt/client/conf:/opt/client/HBase/hbase/lib/*:/opt/client/HBase/hbase/lib/client-facing-thirdparty/*:/opt/client/Hive/Beeline/lib/*

     4. Submit MapReduce jobs. Run the following command to run the sample project:

        yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.MultiComponentExample