Creating a Spark Job

Prerequisites

Upload the dependencies to the corresponding OBS bucket on the Data Management > Package Management page. For details, see Creating a Package.

GUI Description

• Navigation bar on the left

  On the Spark job creation page, the navigation tree on the left contains the Queues and Packages tab pages.

  Table 1 Description of buttons in the left navigation pane

  No.	Tab/Button	Tab/Button Name	Description
  1		Queues	Displays existing queues.
  2		Packages	Displays existing packages.
  3		Create	Create a queue or a package.
  4		Refresh	Refreshes the lists of existing queues and packages.
  5		Search	On the Packages tab page, enter a package name for search.

• Job editing window

  In the job editing window, you can set parameters in Fill Form mode or Write API mode.

  The following uses the Fill Form as an example. In Write API mode, refer to the Data Lake Insight API Reference for parameter settings.

  • Select a Queue: For details about the parameters, see Table 2.

    Table 2 Queue parameters

    Parameter	Description
    Queue	Select the target queue from the drop-down list box.

  • Job Configurations: Refer to Table 3 for details.

    Table 3 Job configuration parameters

    Parameter	Description
    Job Name	Set a job name.
    Application	Select the package to be executed. The value can be .jar or .py.
    Main Class	Enter the name of the main class. When the application type is .jar, the main class name cannot be empty.
    Application Parameters	User-defined parameter. Separate multiple parameters by Enter.
    Spark Arguments	Enter a parameter in the format of key=value. Press Enter to separate multiple key-value pairs.
    JAR Package Dependencies	JAR file on which the Spark job depends
    Python File Dependencies	py-files on which the Spark job depends
    Other Dependencies	Other files on which the Spark job depends
    Group Name	If you select a group when creating a package, you can select all the packages and files in the group. For details about how to create a package, see Creating a Package.
    Retry	Indicates whether to retry a failed job. If you select Yes, you need to set the following parameters: Maximum Retries: Maximum number of retry times. The maximum value is 100.
    Advanced Settings	Skip Configure Select Dependency Resources: For details about the parameters, see Table 4. Configure Resources: For details about the parameters, see Table 5.

    Table 4 Parameters for selecting dependency resources

    Parameter	Description
    Module Name	Dependency modules provided by DLI for executing datasource connection jobs. To access different services, you need to select different modules. CloudTable/MRS HBase: sys.datasource.hbase CloudTable/MRS OpenTSDB: sys.datasource.opentsdb RDS MySQL: sys.datasource.rds RDS PostGre: sys.datasource.rds DWS: sys.datasource.dws CSS: sys.datasource.css
    Resource Package	JAR package on which the Spark job depends.

    Table 5 Resource specification parameters

    Parameter	Description
    Resource Specifications	Select a resource specification from the drop-down list box. The system provides three resource specifications for you to select. The following configuration items in the resource specifications can be modified: Executor Memory Executor Cores Executors Driver Cores Driver Memory If modified, your modified settings of the items are used.
    Executor Memory	Customize the configuration item based on the selected resource specifications.
    Executor Cores	Customize the configuration item based on the selected resource specifications.
    Executors	Customize the configuration item based on the selected resource specifications.
    Driver Cores	Customize the configuration item based on the selected resource specifications.
    Driver Memory	Customize the configuration item based on the selected resource specifications.

    

    Spark job parameter calculation:

    • Number of CUs = Driver Cores + Executors
    • Memory = Driver Memory + (Executors x Executor Memory)

Creating a Spark Job

1. In the Spark job editing window, set related parameters. For details, see the description of the Spark job editing window.
2. Click Execute in the upper right corner of the Spark job editing window to submit the job. The message "The batch job is submitted." is displayed.
3. (Optional) Go to the Spark Jobs page to view the status and logs of the submitted Spark job.
   

   After the job is executed successfully, the job record is saved for only 6 hours.