Creating Queues in an Elastic Resource Pool

Notes and Constraints

• A queue named default is preset in DLI for you to experience. Resources are allocated on demand. You are billed based on the amount of data scanned in each job (unit: GB).
• Queue types:
  • For SQL: Spark SQL jobs can be submitted to SQL queues.
  • For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.

  The queue type cannot be changed. If you want to use another queue type, purchase a new queue.

• The billing mode of a queue cannot be changed.
• The region of a queue cannot be changed.
• Queues with 16 CUs do not support scale-out or scale-in.
• Queues with 64 CUs do not support scale-in.
• When creating a queue, you can only select cross-AZ active-active for yearly/monthly queues and pay-per-use dedicated queues. The price of a cross-AZ queue is twice that of a single-AZ queue.
• Newly created queues need to run jobs before they can be scaled in or out.
• DLI queues cannot access the Internet.

  For how to access the Internet from an elastic resource pool, see Configuring the Connection Between a DLI Queue and a Data Source on the Internet.

Adding a Queue

1. In the navigation pane on the left, choose Resources > Resource Pool.
2. Locate the target elastic resource pool and click Add Queue in the Operation column.
3. On the Add Queue page, configure basic queue information. The following table describes the parameters.

   Table 1 Queue information

   Parameter	Description
   Name	Name of the queue to add
   Type	For SQL: The queue is used to run SQL jobs. General general purpose: The queue is used to run Spark and Flink jobs.
   Engine	If Type is For SQL, the queue engine can be spark or trino.
   Enterprise Project	Select the enterprise project the queue belongs to. Queues under different enterprise projects can be added to an elastic resource pool. Enterprise projects let you manage cloud resources and users by project. For details about how to set enterprise projects, see Enterprise Management User Guide. NOTE: This parameter is displayed only for users who have enabled the Enterprise Management Service.
   Description	Description about the queue.
   Tags	Tags used to identify cloud resources. A tag includes the tag key and tag value. If you want to use the same tag to identify multiple cloud resources, that is, to select the same tag from the drop-down list box for all services, you are advised to create predefined tags on the Tag Management Service (TMS). For details, see Tag Management Service User Guide. NOTE: A maximum of 20 tags can be added. Only one tag value can be added to a tag key. The key name in each resource must be unique. Tag key: Enter a tag key name in the text box. NOTE: A tag key can contain a maximum of 128 characters. Only letters, numbers, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_. Tag value: Enter a tag value in the text box. NOTE: A tag value can contain a maximum of 255 characters. Only letters, numbers, spaces, and special characters (_.:=+-@) are allowed. The value cannot start or end with a space.

4. Click Next. On the displayed page, configure a scaling policy for the queue in the elastic resource pool.
   Figure 1 Configuring a scaling policy when adding a queue
   

   Click Create to add a scaling policy with specified priority, period, Minimum CUs, and Maximum CUs. The parameters of each scaling policy are as follows:

   Table 2 Auto scaling policy

   Parameter	Description
   Priority	Priority of the scaling policy in the current elastic resource pool. A larger value indicates a higher priority. You can set a number ranging from 1 to 100.
   Period	Time segment when the policy takes effects. It can be set only by hour. The start time is on the left, and the end time is on the right. The time range includes the start time but not the end time, that is, [start time, end time). For example, if the Period to 01 and 17, the scaling policy takes effect at 01:00 a.m. till 05:00 p.m. The periods of scaling policies with different priorities cannot overlap.
   Min CU	Minimum number of CUs allowed by the scaling policy. In any time segment of a day, the total minimum CUs of all queues in an elastic resource pool cannot be more than the minimum CUs of the pool. If the minimum CUs of the queue is less than 16 CUs, both Max. Spark Driver Instances and Max. Prestart Spark Driver Instances set in the queue properties do not apply. Refer to Setting Queue Properties.
   Max CU	Maximum number of CUs allowed by the scaling policy. In any time segment of a day, the maximum CUs of any queue in an elastic resource pool cannot be more than the maximum CUs of the pool.

   

   • The first scaling policy is the default policy, and its Period parameter configuration cannot be deleted or modified.
   • Flink jobs cannot trigger automatic scaling of queues in an elastic resource pool.

5. Click OK. View all queues and scaling policies added to the elastic resource pool by referring to Adjusting Scaling Policies for Queues in an Elastic Resource Pool.