Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Ankara Region)/ Using Oozie/ Using Hue to Submit an Oozie Job/ Submitting a Workflow Job/ Submitting a Spark Job
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Submitting a Spark Job

Scenario

Submit an Oozie job of the Spark type on the Hue web UI.

Procedure

  1. Create a workflow. For details, see Creating a Workflow.
  2. On the workflow editing page, select next to Spark program and drag it to the operation area.
  3. In the Spark window that is displayed, set Files, for example, to hdfs://hacluster/user/admin/examples/apps/spark/lib/oozie-examples.jar. Set the value of jar/py name, for example, to org.apache.oozie.example.SparkFileCopy, and click Add.
  4. Set the value of Main class, for example, org.apache.oozie.example.SparkFileCopy.
  5. Click PARAMETER+ to add related input and output parameters.

    For example, add the following parameters:

    • hdfs://hacluster/user/admin/examples/input-data/text/data.txt
    • hdfs://hacluster/user/admin/examples/output-data/spark_workflow

  6. In the Options list text box, specify Spark parameters, for example, --conf spark.yarn.archive=hdfs://hacluster/user/spark/jars/8.1.0.1/spark-archive.zip --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://hacluster/sparkJobHistory.

    The version 8.1.0.1 is used as an example. You can log in to FusionInsight Manager, click in the upper right corner, choose About from the drop-down list, and view the FusionInsight Manager version in the dialog box that is displayed.

  7. Click the configuration button in the upper right corner. Set the value of Spark Master, for example, to yarn-cluster. Set the value of Mode, for example, cluster.
  8. On the configuration page that is displayed, click Delete + to delete a directory, for example, hdfs://hacluster/user/admin/examples/output-data/spark_workflow.
  9. Click PROPERTIES+ and add sharelib used by Oozie. Enter the property name oozie.action.sharelib.for.spark in the left text box and the property value spark in the right text box.
  10. Click in the upper right corner of the Oozie editor.

    If you need to modify the job name before saving the job (default value: My Workflow), click the name directly for modification, for example, Spark-Workflow.

  11. After the configuration is saved, click , and submit the job.

    After the job is submitted, you can view the related contents of the job, such as the detailed information, logs, and processes, on Hue.