Creating a Spark Job

Updated on 2022-07-04 GMT+08:00

View PDF

DLI provides fully-managed Spark computing services by allowing you to execute Spark jobs.

On the Overview page, click Create Job in the upper right corner of the Spark Jobs tab or click Create Job in the upper right corner of the Spark Jobs page. The Spark job editing page is displayed.

Enter the Spark job editing page. A message is displayed, indicating that a temporary DLI data bucket will be created. The created bucket is used to store temporary data generated by DLI, such as job logs and job results. You cannot view job logs if you choose not to create it. You can to periodically delete objects in a bucket or transit objects between different storage classes. The bucket name is set by default.

If you do not need to create a DLI temporary data bucket and do not want to receive this message, select Do not show again and click Cancel.

Prerequisites

Upload the dependencies to the corresponding OBS bucket on the Data Management > Package Management page. For details, see Creating a Package.

GUI Description

Navigation bar on the left

On the Spark job creation page, the navigation tree on the left contains the Queues and Packages tab pages.

**Table 1** Description of buttons in the left navigation pane
No.	Tab/Button Name	Description
1	Queues	Displays existing queues.
2	Packages	Displays existing packages.
3	Create	Create a queue or a package.
4	Refresh	Refreshes the lists of existing queues and packages.
5	Search	On the Packages tab page, enter a package name for search.

Job editing window

In the job editing window, you can set parameters in Fill Form mode or Write API mode.

The following uses the Fill Form as an example. In Write API mode, refer to the Data Lake Insight API Reference for parameter settings.

Select a Queue: For details about the parameters, see Table 2.

**Table 2** Queue parameters
Parameter	Description
Queue	Select the target queue from the drop-down list box.

Job Configurations: Refer to Table 3 for details.

**Table 3** Job configuration parameters
Parameter	Description
Job Name	Set a job name.
Application	Select the package to be executed. The value can be .jar or .py.
Main Class	Enter the name of the main class. When the application type is .jar, the main class name cannot be empty.
Application Parameters	User-defined parameter. Separate multiple parameters by Enter.
Spark Arguments	Enter a parameter in the format of key=value. Press Enter to separate multiple key-value pairs.
JAR Package Dependencies	JAR file on which the Spark job depends
Python File Dependencies	py-files on which the Spark job depends
Other Dependencies	Other files on which the Spark job depends
Group Name	If you select a group when creating a package, you can select all the packages and files in the group. For details about how to create a package, see Creating a Package.
Retry	Indicates whether to retry a failed job. If you select Yes, you need to set the following parameters: Maximum Retries: Maximum number of retry times. The maximum value is 100.
Advanced Settings	Skip Configure Select Dependency Resources: For details about the parameters, see Table 4. Configure Resources: For details about the parameters, see Table 5.

**Table 4** Parameters for selecting dependency resources
Parameter	Description
Module Name	Dependency modules provided by DLI for executing datasource connection jobs. To access different services, you need to select different modules. CloudTable/MRS HBase: sys.datasource.hbase CloudTable/MRS OpenTSDB: sys.datasource.opentsdb RDS MySQL: sys.datasource.rds RDS PostGre: sys.datasource.rds DWS: sys.datasource.dws CSS: sys.datasource.css
Resource Package	JAR package on which the Spark job depends.

**Table 5** Resource specification parameters
Parameter	Description
Resource Specifications	Select a resource specification from the drop-down list box. The system provides three resource specifications for you to select. The following configuration items in the resource specifications can be modified: Executor Memory Executor Cores Executors Driver Cores Driver Memory If modified, your modified settings of the items are used.
Executor Memory	Customize the configuration item based on the selected resource specifications.
Executor Cores	Customize the configuration item based on the selected resource specifications.
Executors	Customize the configuration item based on the selected resource specifications.
Driver Cores	Customize the configuration item based on the selected resource specifications.
Driver Memory	Customize the configuration item based on the selected resource specifications.

NOTE:

Spark job parameter calculation:

Number of CUs = Driver Cores + Executors
Memory = Driver Memory + (Executors x Executor Memory)

Creating a Spark Job

In the Spark job editing window, set related parameters. For details, see the description of the Spark job editing window.
Click Execute in the upper right corner of the Spark job editing window to submit the job. The message "The batch job is submitted." is displayed.
(Optional) Go to the Spark Jobs page to view the status and logs of the submitted Spark job.
NOTE:

After the job is executed successfully, the job record is saved for only 6 hours.