Creating a Spark Job

Prerequisites

• You have uploaded the dependencies to the corresponding OBS bucket on the Data Management > Package Management page. For details, see Creating a Package.
• Before creating a Spark job to access other external data sources, such as OpenTSDB, HBase, Kafka, DWS, RDS, CSS, CloudTable, DCS Redis, and DDS MongoDB, you need to create a cross-source connection to enable the network between the job running queue and external data sources.
  • For details about the external data sources that can be accessed by Spark jobs, see Datasource Connection and Cross-Source Analysis.
  • For details about how to create a datasource connection, see Enhanced Datasource Connections.

    On the Queue Management page, locate the queue you have created, and choose More > Test Address Connectivity in the Operation column to check whether the network connection between the queue and the data source is normal. For details, see Testing Address Connectivity.

GUI Description

• Navigation bar on the left

  On the Spark job creation page, the navigation tree on the left contains the Queues and Packages tab pages.

  Table 1 Description of buttons in the left navigation pane

  No.	Tab/Button	Tab/Button Name	Description
  1		Queues	Display existing queues.
  2		Packages	Display existing packages.
  3	Templates	Templates	Display job templates.
  4		Create	Create a queue or a package.
  5		Refresh	Refresh the lists of existing queues and packages.
  6		Search for packages by name.	On the Packages tab page, enter a package name for search.

• Job editing window

  In the job editing window, you can set parameters in Fill Form mode or Write API mode.

  The following uses the Fill Form as an example. In Write API mode, refer to the Data Lake Insight API Reference for parameter settings.

  • Select a Queue: For details about the parameters, see Table 2.

    Table 2 Queue parameters

    Parameter	Description
    Queues	Select the target queue from the drop-down list box.

  • Job Configurations: Refer to Table 3 for details.

    Table 3 Job configuration parameters

    Parameter	Description
    Job Name	Set a job name.
    Application	Select the package to be executed. The value can be .jar or .py.
    Main Class	Enter the name of the main class. When the application type is .jar, the main class name cannot be empty.
    Application Parameters	User-defined parameters. Separate multiple parameters by Enter. These parameters can be replaced by global variables. For example, if you create a global variable batch_num on the Global Configuration > Global Variables page, you can use {{batch_num}} to replace a parameter with this variable after the job is submitted.
    Spark Arguments	Enter a parameter in the format of key=value. Press Enter to separate multiple key-value pairs. These parameters can be replaced by global variables. For example, if you create a global variable custom_class on the Global Configuration > Global Variables page, you can use "spark.sql.catalog"={{custom_class}} to replace a parameter with this variable after the job is submitted. NOTE: The JVM garbage collection algorithm cannot be customized for Spark jobs.
    Job Type	Type of the Spark image used by a job. This parameter is mandatory when you select a CCE queue. The values are as follows: Basic: Basic images provided by DLI. Select this option for non-AI jobs. AI-enhanced: AI images provided by DLI. Select this option for AI jobs. Image: Custom Spark images. Select an existing image name and version on SWR.
    JAR Package Dependencies	JAR file on which the Spark job depends. You can enter the JAR package name or the corresponding OBS path. The format is as follows: obs://Bucket name/Folder name/Package name.
    Python File Dependencies	py-files on which the Spark job depends. You can enter the Python file name or the corresponding OBS path of the Python file. The format is as follows: obs://Bucket name/Folder name/File name.
    Other Dependencies	Other files on which the Spark job depends. You can enter the name of the dependency file or the corresponding OBS path of the dependency file. The format is as follows: obs://Bucket name/Folder name/File name.
    Group Name	If you select a group when creating a package, you can select all the packages and files in the group. For details about how to create a package, see Creating a Package.
    Access Metadata	Whether to access metadata through Spark jobs. .
    Retry upon Failure	Indicates whether to retry a failed job. If you select Yes, you need to set the following parameters: Maximum Retries: Maximum number of retry times. The maximum value is 100.
    Advanced Settings	Skip Configure Select Dependency Resources: For details about the parameters, see Table 4. Configure Resources: For details about the parameters, see Table 5.

    Table 4 Parameters for selecting dependency resources

    Parameter	Description
    Module Name	Dependency modules provided by DLI for executing datasource connection jobs. To access different services, you need to select different modules. MRS HBase: sys.datasource.hbase DDS: sys.datasource.mongo MRS OpenTSDB: sys.datasource.opentsdb DWS: sys.datasource.dws RDS MySQL: sys.datasource.rds RDS PostGre: sys.datasource.rds DCS: sys.datasource.redis CSS: sys.datasource.css
    Resource Package	JAR package on which the Spark job depends.

    Table 5 Resource specification parameters

    Parameter	Description
    Resource Specifications	Select a resource specification from the drop-down list box. The system provides three resource specifications for you to select. The following configuration items in the resource specifications can be modified: Executor Memory Executor Cores Executors Driver Cores Driver Memory If modified, your modified settings of the items are used.
    Executor Memory	Customize the configuration item based on the selected resource specifications.
    Executor Cores	Customize the configuration item based on the selected resource specifications.
    Executors	Customize the configuration item based on the selected resource specifications.
    Driver Cores	Customize the configuration item based on the selected resource specifications.
    Driver Memory	Customize the configuration item based on the selected resource specifications.

    

    Spark job parameter calculation:

    • Number of CUs = Number of driver CPU cores + Number of executors x Number of executor CPU cores

      The cluster management plane and driver use some CU resources. Number of Executors * Number of Executor Cores must be smaller than the number of computing CUs of the queue.

    • Memory = Driver memory + (Number of Executors x Executor memory)

Creating a Spark Job

1. In the Spark job editing window, set related parameters. For details, see the description of the Spark job editing window.
2. Click Execute in the upper right corner of the Spark job editing window to submit the job. The message "The batch job is submitted." is displayed.
3. (Optional) Go to the Spark Jobs page to view the status and logs of the submitted Spark job.