Updated on 2022-07-11 GMT+08:00

Development Procedure

  1. Analyze the service.

    1. Analyze and process logs using Mapreduce in the client example directory.
    2. Move Mapreduce analysis results to the data analysis result directory, and set the data file access permission to 660.
    3. To analyze data every day, perform 1.a and 1.b every day.

  2. Implement the service.

    1. Log in to the node where the client is located, and create the dataLoad directory, for example, /opt/client/Oozie/oozie-client-*/examples/apps/dataLoad/. This directory is used as a program running directory to store files that are edited subsequently.

      You can directly copy the content in the map-reduce directory of the example directory to the dataLoad directory and edit the content.

      Replace oozie-client-* in the directory with the actual version number.

    2. Compile a workflow job property file job.properties.

      For details, see job.properties.

    3. Compile a workflow job using workflow.xml.
      Table 1 Actions in a Workflow

      No.

      Procedure

      Description

      1

      Define the start action.

      For details, see Start Action

      2

      Define the MapReduce action.

      For details, see MapReduce Action

      3

      Define the FS action.

      For details, see FS Action

      4

      Define the end action.

      For details, see End Action

      5

      Define the kill action.

      For details, see Kill Action

      Dependent or newly developed JAR packages must be saved in dataLoad/lib.

      The following provides an example workflow file:

      <workflow-app xmlns="uri:oozie:workflow:1.0" name="data_load"> 
         <start to="mr-dataLoad"/> 
         <action name="mr-dataLoad"> 
             <map-reduce> <resource-manager>${resourceManager}</resource-manager>  
                  <name-node>${nameNode}</name-node> 
                  <prepare> 
                      <delete path="${nameNode}/user/${wf:user()}/${dataLoadRoot}/output-data/map-reduce"/> 
                  </prepare> 
                  <configuration> 
                      <property> 
                          <name>mapred.job.queue.name</name> 
                          <value>${queueName}</value> 
                      </property> 
                      <property> 
                          <name>mapred.mapper.class</name> 
                          <value>org.apache.oozie.example.SampleMapper</value> 
                      </property> 
                      <property> 
                          <name>mapred.reducer.class</name> 
                          <value>org.apache.oozie.example.SampleReducer</value> 
                      </property> 
                      <property> 
                          <name>mapred.map.tasks</name> 
                          <value>1</value> 
                      </property> 
                      <property> 
                          <name>mapred.input.dir</name> 
                          <value>/user/oozie/${dataLoadRoot}/input-data/text</value> 
                      </property> 
                      <property> 
                          <name>mapred.output.dir</name> 
                          <value>/user/${wf:user()}/${dataLoadRoot}/output-data/map-reduce</value> 
                      </property> 
                  </configuration> 
              </map-reduce> 
              <ok to="copyData"/> 
              <error to="fail"/> 
          </action> 
       
          <action name="copyData"> 
               <fs> 
                   <delete path='${nameNode}/user/oozie/${dataLoadRoot}/result'/> 
                   <move source='${nameNode}/user/${wf:user()}/${dataLoadRoot}/output-data/map-reduce'  
                         target='${nameNode}/user/oozie/${dataLoadRoot}/result'/> 
                   <chmod path='${nameNode}/user/oozie/${dataLoadRoot}/result' permissions='-rwxrw-rw-' dir-files='true'></chmod> 
              </fs> 
              <ok to="end"/> 
              <error to="fail"/> 
          </action> 
       
          <kill name="fail"> 
              <message>This workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
          </kill> 
          <end name="end"/> 
      </workflow-app> 
    4. Compile a Coordinator job using coordinator.xml.

      The Coordinator job is used to analyze data every day. For details, see coordinator.xml.

  3. Upload the workflow file.

    1. Use or switch to the user account that is granted with rights to upload files to the HDFS. For details about developer account preparation, see Preparing Development and Operating Environment.
    2. Implement Kerberos authentication for the user account.
    3. Run the HDFS upload command to upload the dataLoad folder to a specified directory on the HDFS (user developuser must have the read/write permission for the directory).

      The specified directory must be the same as oozie.coord.application.path and workflowAppUri defined in job.properties.

  4. Execute the workflow file.

    1. Log in to the client node, implement Kerberos authentication for user developuser.

      cd /opt/client

      source bigdata_env

      kinit developuser

    2. Run the following command to start the workflow:

    Command:

    oozie job -oozie https://oozie server hostname https://oozie server hostname:port/oozie -config job.propertiesfile path -run

    Parameter list:

    Table 2 Parameters

    Parameter

    Description

    job

    Indicates that a job is to be executed.

    -oozie

    Indicates the (any instance) Oozie server address.

    -config

    Indicates the path of job.properties.

    -run

    Indicates the starts workflow.

    For example:

    oozie job -oozie https://10-1-130-10:21003/oozie -configjob.properties -run