Compiling and Running Applications

Scenario

You can run applications in the Linux environment after application code development is complete.

Prerequisites

• You have stored the configuration file obtained in Preparing the Configuration Files for Connecting to the Cluster in the conf directory, for example, /opt/client/conf. This directory must be the same as the directory where the MRTest-XXX.jar package uploaded in 2 is stored.
• You have prepared the data to be processed by referring to Planning MapReduce Accessing Multi-Component Sample Program Data.

Procedure

1. Go to the local root directory of the sample project and run the following command in the Windows CLI to package the sample project:

   mvn -s "{maven_setting_path}" clean package

   

   • In the preceding command, {maven_setting_path} indicates the path of the settings.xml file of the local Maven.
   • After the packaging is successful, obtain the JAR package, for example, MRTest-XXX.jar, from the target subdirectory in the root directory of the project. The JAR package name is subject to the actual packaging result.

2. Upload the generated application package MRTest-XXX.jar to the Linux client, for example, /opt/client/conf, in the same directory as the configuration file.
3. Execute the sample project on Linux.
   • For the MapReduce statistics sample project, run the following command to configure parameters and submit jobs.

     yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath>

     This command is used to set parameters and submit jobs. <inputPath> indicates the input path in HDFS and <outputPath> indicates the output path in HDFS.

     

     • Before running the yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.FemaleInfoCollector <inputPath> <outputPath&gt command, ensure that the <outputPath> directory does not exist. Otherwise, an error is reported.
     • Do not restart the HDFS service during the running of MapReduce jobs. Otherwise, the jobs may fail.
   • For the sample application about multi-component access from MapReduce, perform the following steps.
     1. Create the jaas_mr.conf file in the /opt/client/conf folder. The file content is as follows:

        Client {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        keyTab="user.keytab"
        principal="test@<System domain name>"
        useTicketCache=false
        storeKey=true
        debug=true;
        };

        

        • test: indicates the name of the user created in User Information for Cluster Authentication, for example, developuser.
        • System domain name: Log in to FusionInsight Manager, choose System > Permission > Domain and Mutual Trust, and view the value of Local Domain, which is the current system domain name.
     2. Add the classpath required for running the sample project in the Linux environment. For example:

        export YARN_USER_CLASSPATH=/opt/client/conf:/opt/client/HBase/hbase/lib/*:/opt/client/HBase/hbase/lib/client-facing-thirdparty/*:/opt/client/Hive/Beeline/lib/*

     3. Submit the MapReduce job and run the following command to run the sample project.

        yarn jar MRTest-XXX.jar com.huawei.bigdata.mapreduce.examples.MultiComponentExample