Creating and Submitting a Spark Jar Job

Step 1: Logging in to the Cloud

To use DLI, you need to log in to the cloud.

Step 2: Uploading Data to OBS

Write a Spark Jar job program , and compile and package it as spark-examples.jar. Perform the following steps to submit the job:

Before submitting Spark Jar jobs, upload data files to OBS.

1. In the services displayed, click Object Storage Service (OBS) in Storage.
2. The OBS console page is displayed.
3. Create a bucket. The bucket name must be globally unique. In this example, assume that the bucket name is dli-test-obs01.
   1. Click Create Bucket.
   2. On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or set them as required.
   3. Click Create Now.
4. Click dli-test-obs01 to switch to the Overview page.
5. From the left navigation tree, click Objects. Click Upload Object. In the displayed dialog box, drag files or folders to upload or add file to the file upload box, for example, spark-examples.jar. Then, click Upload.

   After the file is uploaded successfully, the file path is obs://dli-test-obs01/spark-examples.jar.

   For more information about OBS operations, see the Object Storage Service Console Operation Guide.

   For more information about the tool, see the OBS Tool Guide.

   

   You are advised to use an OBS tool, such as OBS Browser+, to upload large files because OBS Console has restrictions on the file size and quantity.

   • OBS Browser+ is a graphical tool that provides complete functions for managing your buckets and objects in OBS. You are advised to use this tool to create buckets or upload objects.

Step 3: Logging In to the DLI Management Console

To submit Spark jobs, you need to enter the Spark job creation page first.

1. In the list, choose .
2. The DLI management console page is displayed.
   

   If you log in to the DLI management console for the first time, you need to be authorized to access OBS. For details, see "Service Authorization".

Step 4: Creating a Queue

If you submit a Spark job for the first time, you need to create a queue first. For example, create a queue named sparktest and set Queue Type to General Queue.

1. In the navigation pane of the DLI management console, choose Queue Management.
2. In the upper right corner of the Queue Management page, click Create Queue to create a queue.
3. Create a queue, name it sparktest, and set the queue usage to for general purpose. For details, see Creating a Queue.

4. Click Create Now to create a queue.

Step 5: Creating a Package

Before submitting a Spark job, you need to create a package, for example, spark-examples.jar.

1. In the navigation pane on the left of the DLI console, choose Data Management > Package Management.
2. On the Package Management page, click Create in the upper right corner to create a package.
3. In the Create Package dialog box, set Type to JAR, OBS Path to the path of the spark-examples.jar package in Step 2: Uploading Data to OBS, and Group to Do not use.

4. Click OK.

   You can view and select the package on the Package Management page.

For details about how to create a package, see "Creating a Package".

Step 6: Submitting a Spark Job

1. On the DLI management console, choose Job Management > Spark Jobs in the navigation pane on the left. On the displayed page, click Create Job.
2. On the Spark job editing page, set Queues to the queue created in Step 4: Creating a Queue and Application to the package created in Step 5: Creating a Package.

   For details about other parameters, see the description of the Spark job editing page in "Creating a Spark Job".

3. Click Execute in the upper right corner of the Spark job editing window, read and agree to the privacy agreement, and click OK. Submit the job. A message is displayed, indicating that the job is submitted.

4. (Optional) Switch to the Job Management > Spark Jobs page to view the status and logs of the submitted Spark job.
   

   When you click Execute on the DLI management console for the first time, you need to read the privacy agreement. Once agreed to the agreement, you will not receive any privacy agreement messages for subsequent operations.