Help Center/ Data Lake Insight/ User Guide/ Creating an Elastic Resource Pool and Queues Within It/ Managing Queues/ Setting Queue Properties
Updated on 2025-03-04 GMT+08:00
View PDF
Share

Setting Queue Properties

Scenario

DLI allows you to set properties for queues.

You can currently set the following property parameters:

  • Spark driver parameters: Set them to improve the scheduling efficiency of queues.
  • Set Result Saving Policy: Set it to determine whether to save job results for queues to a DLI job bucket.

This section describes how to set queue properties on the management console.

Notes and Constraints

  • Only SQL queues of the Spark engine support configuring queue properties.
  • Setting queue properties is only supported after the queue has been created.
  • Queue properties cannot be set in batches.
  • For a queue in an elastic resource pool, if the minimum number of CUs of the queue is less than 16 CUs, both Max. Spark Driver Instances and Max. Prestart Spark Driver Instances set in the queue properties do not apply.
  • After the Set Result Saving Policy is enabled, that is, job results are saved to a DLI job bucket, you must configure the DLI job bucket before submitting SQL jobs. Failure to do so may result in SQL jobs not being submitted successfully.

Procedure

  1. In the navigation pane of the DLI management console, choose Resources > Queue Management.
  2. Locate the queue for which you want to set properties, click More in the Operation column, and select Set Property.
  3. Go to the queue property setting page and set property parameters. Table 1 describes the property parameters.
    Table 1 Queue properties

    Property Type

    Property

    Description

    Value Range

    Spark driver type

    Max. Spark Driver Instances

    Maximum number of Spark drivers can be started on this queue, including the Spark driver that is prestarted and the Spark driver that runs jobs.
    • For a 16-CU queue, the value is 2.
    • For a queue that has more than 16 CUs, the value range is [2, queue CUs/16].
    • If the minimum number of CUs of the queue is less than 16 CUs, this configuration item does not apply.

    Max. Prestart Spark Driver Instances

    Maximum number of Spark drivers can be prestarted on this queue. When the number of Spark drivers that run jobs exceeds the value of Max. Concurrency per Instance, the jobs are allocated to the Spark drivers that are prestarted.
    • For a 16-CU queue, the value range is 0 to 1.
    • For a queue that has more than 16 CUs, the value range is [2, queue CUs/16].
    • If the minimum number of CUs of the queue is less than 16 CUs, this configuration item does not apply.

    Max. Concurrency per Instance

    Maximum number of jobs can be concurrently executed by a Spark driver. When the number of jobs exceeds the value of this parameter, the jobs are allocated to other Spark drivers.

    1–32

    Job result saving policy

    Set Result Saving Policy

    Whether to save job results to a DLI job bucket.

    This parameter is only available for Spark SQL queues.

    Once enabled, this feature cannot be disabled, and job results of a queue are consistently saved to the DLI job bucket you configure.

    Before enabling this feature, make sure you have configured a DLI job bucket. For how to configure a DLI job bucket, see Configuring a DLI Job Bucket.

    To check if you have enabled the function to save SQL job results to a DLI job bucket, refer to How Do I Check if Job Result Saving to a DLI Job Bucket Is Enabled for a SQL Queue?

    You are advised to enable the feature of saving job results to DLI job buckets to more effectively manage and store SQL job query results.

    N/A
  4. Click OK.

How Do I Check if Job Result Saving to a DLI Job Bucket Is Enabled for a SQL Queue?

  • Method 1: View the result path on the SQL job details page.
    1. Log in to the DLI console. In the navigation pane on the left, choose Job Management > SQL Jobs.
    2. Locate a desired SQL job and click next to the queue name to expand the job details.
    3. Check the result path in the job details.
      • If the result path displays a custom DLI job bucket, the feature of saving job results to a DLI job bucket is enabled for the queue running the job.
      • If the result path is not displayed in the job details, the feature is not enabled for the queue running the job.
  • Method 2: Check whether the feature of saving job results to a job bucket is enabled in the SQL queue properties.
    1. Log in to the DLI management console. In the navigation pane on the left, choose Resources > Queue Management.
    2. Locate the queue for which you want to set properties, click More in the Operation column, and select Set Property.
    3. On the Set Property dialog box that appears, check whether the feature is enabled.
Parent topic: Managing Queues