Developing a DLI Spark Job

Updated on 2022-08-17 GMT+08:00

View PDF

This section introduces how to develop a DLI Spark job on DataArts Factory.

Scenario Description

In most cases, SQL is used to analyze and process data when using Data Lake Insight (DLI). However, SQL is usually unable to deal with complex processing logic. In this case, Spark jobs can help. This section uses an example to demonstrate how to submit a Spark job on DataArts Factory.

The general submission procedure is as follows:

Create a DLI cluster and run a Spark job using physical resources of the DLI cluster.
Obtain a demo JAR package of the Spark job and associate with the JAR package on DataArts Factory.
Create a DataArts Factory job and submit it using the DLI Spark node.

Preparations

Object Storage Service (OBS) has been enabled and a bucket, for example, obs://dlfexample, has been created for storing the JAR package of the Spark job.
DLI has been enabled, and the Spark cluster spark_cluster has been created for providing physical resources required for the Spark job.

Obtaining Spark Job Code

The Spark job code used in this example comes from the maven repository that can be download from https://repo.maven.apache.org/maven2/org/apache/spark/spark-examples_2.10/1.1.1/spark-examples_2.10-1.1.1.jar. This Spark job is to calculate the approximate value of π.

After obtaining the JAR package of the Spark job codes, upload it to the OBS bucket. The save path is obs://dlfexample/spark-examples_2.10-1.1.1.jar.
Log in to the DataArts Studio console. Locate an instance and click Access. On the displayed page, locate a workspace and click DataArts Factory.

Figure 1 DataArts Factory
In the navigation tree on the left, choose Configuration > Manage Resource. Click Create Resource and create resource spark-example on DataArts Factory and associate it with the JAR package obtained in 1.

Figure 2 Creating a resource

Submitting a Spark Job

You need to create a job on DataArts Factory and submit the Spark job using the DLI Spark node of the job.

Create a job named job_DLI_Spark for the DataArts Factory module.

Figure 3 Creating a job
Go to the job development page, drag the DLI Spark node to the canvas, and click the node to configure node properties.

Figure 4 Configuring node properties

Description of key properties:
- DLI Cluster Name: name of the Spark cluster created in DLI
- Job Running Resource: Maximum CPU and memory resources that can be used when a DLI Spark node is running.
- Major Job Class: major class of a DLI Spark node. In this example, the major class is org.apache.spark.examples.SparkPi.
- JAR Package: Resource created in 3.
After the job orchestration is complete, click to test the job.

Figure 5 Job logs (for reference only)
If no error is recorded in logs, save and submit the job.

Parent topic: Usage Guidance

Previous topic: Developing a Hive SQL Job

Next topic: Developing an MRS Flink Job

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot