On this page

Show all

Help Center/ Data Lake Insight/ User Guide (ME-Abu Dhabi Region)/ Job Management/ Spark Job Management/ Creating a Spark Job

Creating a Spark Job

Updated on 2022-12-07 GMT+08:00

View PDF

DLI provides fully-managed Spark computing services by allowing you to execute Spark jobs.

On the Overview page, click Create Job in the upper right corner of the Spark Jobs tab or click Create Job in the upper right corner of the Spark Jobs page. The Spark job editing page is displayed.

On the Spark job editing page, a message is displayed, indicating that a temporary DLI data bucket will be created. The created bucket is used to store temporary data generated by DLI, such as job logs and job results. You cannot view job logs if you choose not to create it. The bucket will be created and the default bucket name is used.

If you do not need to create a DLI temporary data bucket and do not want to receive this message, select Do not show again and click Cancel.

Prerequisites

You have uploaded the dependencies to the corresponding OBS bucket on the Data Management > Package Management page. For details, see Creating a Package.
Before creating a Spark job to access other external data sources, such as OpenTSDB, HBase, Kafka, DWS, RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to enable the network between the job running queue and external data sources.
- For details about the external data sources that can be accessed by Spark jobs, see Datasource Connection and Cross-Source Analysis.
- For details about how to create a datasource connection, see Enhanced Datasource Connections.
  On the Queue Management page, locate the queue you have created, and choose More > Test Address Connectivity in the Operation column to check whether the network connection between the queue and the data source is normal. For details, see Testing Address Connectivity.

GUI Description

Navigation bar on the left

On the Spark job creation page, the navigation tree on the left contains the Queues and Packages tab pages.

**Table 1** Description of buttons in the left navigation pane
No.	Tab/Button	Tab/Button Name	Description
1		Queues	Display existing queues.
2		Packages	Display existing packages.
3	Templates	Templates	Display job templates.
4		Create	Create a queue or a package.
5		Refresh	Refresh the lists of existing queues and packages.
6		Search for packages by name.	On the Packages tab page, enter a package name for search.

Job editing window

In the job editing window, you can set parameters in Fill Form mode or Write API mode.

The following uses the Fill Form as an example. In Write API mode, refer to the Data Lake Insight API Reference for parameter settings.

Select a Queue: For details about the parameters, see Table 2.

**Table 2** Queue parameters
Parameter	Description
Queues	Select the target queue from the drop-down list box.

Job Configurations: Refer to Table 3 for details.

**Table 3** Job configuration parameters
Parameter	Description
Job Name	Set a job name.
Application	Select the package to be executed. The value can be .jar or .py.
Main Class	Enter the name of the main class. When the application type is .jar, the main class name cannot be empty.
Application Parameters	User-defined parameters. Separate multiple parameters by Enter. These parameters can be replaced by global variables. For example, if you create a global variable batch_num on the Global Configuration > Global Variables page, you can use {{batch_num}} to replace a parameter with this variable after the job is submitted.
Spark Arguments	Enter a parameter in the format of key=value. Press Enter to separate multiple key-value pairs. These parameters can be replaced by global variables. For example, if you create a global variable custom_class on the Global Configuration > Global Variables page, you can use "spark.sql.catalog"={{custom_class}} to replace a parameter with this variable after the job is submitted. NOTE: The JVM garbage collection algorithm cannot be customized for Spark jobs.
Job Type	Type of the Spark image used by a job. This parameter is mandatory when you select a CCE queue. The values are as follows: Basic: Basic images provided by DLI. Select this option for non-AI jobs. AI-enhanced: AI images provided by DLI. Select this option for AI jobs. Image: Custom Spark images. Select an existing image name and version on SWR.
JAR Package Dependencies	JAR file on which the Spark job depends. You can enter the JAR package name or the corresponding OBS path. The format is as follows: obs://Bucket name/Folder name/Package name.
Python File Dependencies	py-files on which the Spark job depends. You can enter the Python file name or the corresponding OBS path of the Python file. The format is as follows: obs://Bucket name/Folder name/File name.
Other Dependencies	Other files on which the Spark job depends. You can enter the name of the dependency file or the corresponding OBS path of the dependency file. The format is as follows: obs://Bucket name/Folder name/File name.
Group Name	If you select a group when creating a package, you can select all the packages and files in the group. For details about how to create a package, see Creating a Package.
Access Metadata	Whether to access metadata through Spark jobs. .
Retry upon Failure	Indicates whether to retry a failed job. If you select Yes, you need to set the following parameters: Maximum Retries: Maximum number of retry times. The maximum value is 100.
Advanced Settings	Skip Configure Select Dependency Resources: For details about the parameters, see Table 4. Configure Resources: For details about the parameters, see Table 5.

**Table 4** Parameters for selecting dependency resources
Parameter	Description
Module Name	Dependency modules provided by DLI for executing datasource connection jobs. To access different services, you need to select different modules. MRS HBase: sys.datasource.hbase DDS: sys.datasource.mongo MRS OpenTSDB: sys.datasource.opentsdb DWS: sys.datasource.dws RDS MySQL: sys.datasource.rds RDS PostGre: sys.datasource.rds DCS: sys.datasource.redis CSS: sys.datasource.css
Resource Package	JAR package on which the Spark job depends.

**Table 5** Resource specification parameters
Parameter	Description
Resource Specifications	Select a resource specification from the drop-down list box. The system provides three resource specifications for you to select. The following configuration items in the resource specifications can be modified: Executor Memory Executor Cores Executors Driver Cores Driver Memory If modified, your modified settings of the items are used.
Executor Memory	Customize the configuration item based on the selected resource specifications.
Executor Cores	Customize the configuration item based on the selected resource specifications.
Executors	Customize the configuration item based on the selected resource specifications.
Driver Cores	Customize the configuration item based on the selected resource specifications.
Driver Memory	Customize the configuration item based on the selected resource specifications.

NOTE:

Spark job parameter calculation:

Number of CUs = Number of driver CPU cores + Number of executors x Number of executor CPU cores
The cluster management plane and driver use some CU resources. Number of Executors * Number of Executor Cores must be smaller than the number of computing CUs of the queue.
Memory = Driver memory + (Number of Executors x Executor memory)

Creating a Spark Job

In the Spark job editing window, set related parameters. For details, see the description of the Spark job editing window.
Click Execute in the upper right corner of the Spark job editing window to submit the job. The message "The batch job is submitted." is displayed.
(Optional) Go to the Spark Jobs page to view the status and logs of the submitted Spark job.

Parent topic: Spark Job Management

Previous topic: Spark Job Management

Next topic: Queue Management

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot