Development Procedure

1. Analyze and process logs using Mapreduce in the client example directory.
2. Move Mapreduce analysis results to the data analysis result directory, and set the data file access permission to 660.
3. To analyze data every day, perform 1.a and 1.b every day.

1. Log in to the node where the client is located, and create the dataLoad directory, for example, /opt/client/Oozie/oozie-client-*/examples/apps/dataLoad/. This directory is used as a program running directory to store files that are edited subsequently.
   

   You can directly copy the content in the map-reduce directory of the example directory to the dataLoad directory and edit the content.

   Replace oozie-client-* in the directory with the actual version number.

2. Compile a workflow job property file job.properties.

   For details, see job.properties.

3. Compile a workflow job using workflow.xml.

   Table 1 Actions in a Workflow

   No.	Procedure	Description
   1	Define the start action.	For details, see Start Action
   2	Define the MapReduce action.	For details, see MapReduce Action
   3	Define the FS action.	For details, see FS Action
   4	Define the end action.	For details, see End Action
   5	Define the kill action.	For details, see Kill Action

   

   Dependent or newly developed JAR packages must be saved in dataLoad/lib.

   The following provides an example workflow file:

   <workflow-app xmlns="uri:oozie:workflow:1.0" name="data_load"> 
      <start to="mr-dataLoad"/> 
      <action name="mr-dataLoad"> 
          <map-reduce> <resource-manager>${resourceManager}</resource-manager>  
               <name-node>${nameNode}</name-node> 
               <prepare> 
                   <delete path="${nameNode}/user/${wf:user()}/${dataLoadRoot}/output-data/map-reduce"/> 
               </prepare> 
               <configuration> 
                   <property> 
                       <name>mapred.job.queue.name</name> 
                       <value>${queueName}</value> 
                   </property> 
                   <property> 
                       <name>mapred.mapper.class</name> 
                       <value>org.apache.oozie.example.SampleMapper</value> 
                   </property> 
                   <property> 
                       <name>mapred.reducer.class</name> 
                       <value>org.apache.oozie.example.SampleReducer</value> 
                   </property> 
                   <property> 
                       <name>mapred.map.tasks</name> 
                       <value>1</value> 
                   </property> 
                   <property> 
                       <name>mapred.input.dir</name> 
                       <value>/user/oozie/${dataLoadRoot}/input-data/text</value> 
                   </property> 
                   <property> 
                       <name>mapred.output.dir</name> 
                       <value>/user/${wf:user()}/${dataLoadRoot}/output-data/map-reduce</value> 
                   </property> 
               </configuration> 
           </map-reduce> 
           <ok to="copyData"/> 
           <error to="fail"/> 
       </action> 
    
       <action name="copyData"> 
            <fs> 
                <delete path='${nameNode}/user/oozie/${dataLoadRoot}/result'/> 
                <move source='${nameNode}/user/${wf:user()}/${dataLoadRoot}/output-data/map-reduce'  
                      target='${nameNode}/user/oozie/${dataLoadRoot}/result'/> 
                <chmod path='${nameNode}/user/oozie/${dataLoadRoot}/result' permissions='-rwxrw-rw-' dir-files='true'></chmod> 
           </fs> 
           <ok to="end"/> 
           <error to="fail"/> 
       </action> 
    
       <kill name="fail"> 
           <message>This workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
       </kill> 
       <end name="end"/> 
   </workflow-app> 

4. Compile a Coordinator job using coordinator.xml.

   The Coordinator job is used to analyze data every day. For details, see coordinator.xml.

1. Use or switch to the user account that is granted with rights to upload files to the HDFS. For details about developer account preparation, see Preparing Development and Operating Environment.
2. Implement Kerberos authentication for the user account.
3. Run the HDFS upload command to upload the dataLoad folder to a specified directory on the HDFS (user developuser must have the read/write permission for the directory).
   

   The specified directory must be the same as oozie.coord.application.path and workflowAppUri defined in job.properties.

1. Log in to the client node, implement Kerberos authentication for user developuser.

   cd /opt/client

   source bigdata_env

   kinit developuser

2. Run the following command to start the workflow:

Command:

oozie job -oozie https://oozie server hostname:port/oozie -config job.propertiesfile path -run

Parameter list:

For example:

oozie job -oozie https://10-1-130-10:21003/oozie -config job.properties -run