Overview
Queue
Queues in DLI are computing resources, which are the basis for using DLI. All executed jobs require computing resources.
Currently, DLI provides two types of queues: For SQL and For general purpose.
- For SQL: The queue is used to run SQL jobs.
- For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.
Constraints
- A queue named default is preset in DLI for you to experience. Resources are allocated on demand. You are billed based on the amount of data scanned in each job (unit: GB).
- Queue types:
- For SQL: Spark SQL jobs can be submitted to SQL queues.
- For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.
The queue type cannot be changed. If you want to use another queue type, purchase a new queue.
- The billing mode of a queue cannot be changed.
- The region of a queue cannot be changed.
- Queues with 16 CUs do not support scale-out or scale-in.
- Queues with 64 CUs do not support scale-in.
- When creating a queue, you can only select cross-AZ active-active for yearly/monthly queues and pay-per-use dedicated queues. The price of a cross-AZ queue is twice that of a single-AZ queue.
- A newly created queue can be scaled in or out only after a job is executed on the queue.
- DLI queues cannot access the Internet.
For details about how to access the Internet from an elastic resource pool, see Configuring the Connection Between a DLI Queue and a Data Source on the Internet.
Difference Between Computing and Storage Resources
Resource |
How to Obtain |
Billing Mode |
Function |
---|---|---|---|
Compute resource |
Create queue on the DLI management console. |
Billed based on the amount of data scanned or CUHs. |
Used for executing queries. |
Storage resource |
DLI has a 5 GB quota. |
Billed based on the amount of stored data. |
Used for storing data in the database and DLI tables. |
- Storage resources are internal storage resources of DLI for storing database and DLI tables and represent the amount of data stored in DLI.
- By default, DLI provides a 5 GB quota for storage resources. If you need over 5 GB storage resources, submit a service ticket to apply for more storage resources.
- A queue named default is preset in DLI. DLI adopts the pay-per-use billing mode. If you are uncertain about the required queue capacity or have no available queue capacity to run queries, you can execute jobs using this queue.
- The default queue is used only for user experience. It may be occupied by multiple users at a time. Therefore, it is possible that you fail to obtain the resource for related operations. You are advised to use a self-built queue to execute jobs.
Dedicated Queue
Resources of a dedicated queue are not released when the queue is idle. That is, resources are reserved regardless of whether the queue is used. Dedicated queues ensure that resources exist when jobs are submitted. When purchasing a pay-per-use queue, you can select a dedicated queue. Dedicated queues are billed for 24 hours regardless of whether they are used.
Cross-AZ Queues
An AZ contains one or more physical data centers. Each AZ has independent cooling, fire extinguishing, moisture-proofing, and electricity facilities. Within an AZ, computing, network, storage, and other resources are logically divided into multiple clusters. AZs within a region are interconnected using high-speed optical fibers to allow you to build cross-AZ high-availability systems. For more information, see Regions and AZs.
DLI dual-AZ queues improve data availability by creating a duplicate queue in the second AZ. You can continuously use DLI when one AZ is unavailable. The dual-AZ mode suits those who require high availability.
A DLI dual-AZ queue is created with the same compute resources in two AZs. For example, if you require 1,400 CUs, you can select 1,400 CUs and select the dual-AZ option when creating a queue. Then, DLI creates 1,400 CUs dedicated compute resources in two AZs. When one AZ is unavailable, the other AZ can properly process your compute tasks.
- Currently, dual-AZ mode is supported only for pay-per-use dedicated queues. The default queue and the normal pay-per-use queues are not supported.
- If you select Dual-AZ when purchasing a queue, the billing is twice as that in single-AZ mode.
Elastic Scaling (Pay-per-Use Queues)
DLI allows you to flexibly scale in or out pay-per-use queues on demand. After a pay-per-use queue with specified specifications is created, you can scale it in and out as required.
To change the queue specifications, see Elastic Scaling. Currently, only pay-per-use queues can be scaled. After scaling, you are still charged based on the usage duration of CUs.
Scaling can be performed for a newly created queue only when jobs are running on this queue.
Scheduled Elastic Scaling
DLI allows you to schedule tasks for periodic queue scaling. After creating a queue, the scheduled scaling tasks can be executed.
- After scaling, you are still charged based on the usage duration of CUs. Currently, scheduled scaling tasks are available only for a queue with more than 64 CUs. That is, the minimum number of CUs in a queue is 64.
- Yearly/monthly queues can be scaled out through scheduled tasks only. The fees cover the prepaid yearly/monthly package and the expanded pay-per-use CUs. In other words, your will be billed on a pay-per-use basis for the resources not included in the yearly/monthly package. Currently, scheduled scaling tasks are available only for a yearly/monthly queue with more than 64 CUs.
Scaling can be performed for a newly created queue only when jobs are running on this queue.
Automatic Queue Scaling
Flink jobs use pay-per-use queues. DLI can automatically trigger scaling for jobs based on the job size.
Scaling can be performed for a newly created queue only when there are jobs running on this queue.
Queue Management Page
Queue Management provides the following functions:
- Managing Permissions
- Creating a Queue
- Deleting a Queue
- Modifying CIDR Block
- Elastic Scaling
- Scheduling CU Changes
- Testing Address Connectivity
- Creating a Topic for Key Event Notifications
- Managing Queue Tags
To receive notifications when a DLI job fails, SMN Administrator permissions are required.
The queue list displays all queues created by you and the default queue. You can view information about queues, such as the queue capacity, billing mode, and more. Queues are listed in chronological order by default in the queue list, with the most recently created queues displayed at the top.
Parameter |
Description |
---|---|
Name |
Name of a queue. |
Type |
Queue type.
|
Specifications |
Queue size. Unit: CU
CU is the pricing unit of queues. A CU consists of 1 vCPU and 4-GB memory. The computing capabilities of queues vary with queue specifications. The higher the specifications, the stronger the computing capability. |
Actual CUs |
Actual size of the current queue. |
Elastic Scaling |
Target CU value for scheduled scaling, or the maximum and minimum CU values of the current specifications. |
Billing Mode |
|
Username |
Queue owner |
Enterprise Project |
Displays the enterprise project to which the created queue belongs. If the project is not an enterprise project, -- is displayed. An enterprise project facilitates project-level management and grouping of cloud resources and users. For details about how to set enterprise projects, see Enterprise Center Overview.
NOTE:
This parameter is displayed only for users who have enabled the Enterprise Management Service. |
Description |
Description of a queue specified during queue creation. If no description is provided, -- is displayed. |
Operation |
|
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot