On this page

Queue
Difference Between Computing and Storage Resources
Dedicated Queue
Cross-AZ Queues
AI-enhanced queue
Modifying Queue Specifications (Scaling)
Queue Management Page

Show all

Help Center/ Data Lake Insight/ User Guide (Paris Region)/ Queue Management/ Overview

Overview

Updated on 2022-07-04 GMT+08:00

View PDF

Queue

Queues in DLI are computing resources, which are the basis for using DLI. All executed jobs require computing resources.

Currently, DLI provides two types of queues, For SQL and For general use. SQL queues are used to run SQL jobs. General-use queues are compatible with Spark queues of earlier versions and are used to run Spark and Flink jobs.

NOTE:

The SQL queue automatically restarts every 12 hours.

Difference Between Computing and Storage Resources

**Table 1** Difference between computing and storage resources
Resource	How to Obtain	Function
Compute resource	Create queue on the DLI management console.	Used for executing queries.
Storage resource	DLI has a 5 GB quota.	Used for storing data in the database and DLI table.

NOTE:

Storage resources are internal storage resources of DLI and represent the amount of data stored in DLI.
By default, DLI provides a 5 GB quota for storage resources.
A queue named default is preset in DLI. If you are uncertain about the required queue capacity or have no available queue capacity to run queries, you can execute jobs using this queue.

The default queue is used only for user experience. It may be occupied by multiple users at a time. Therefore, it is possible that you fail to obtain the resource for related operations. You are advised to use a self-built queue to execute jobs.

Dedicated Queue

Resources of a dedicated queue are not released when the queue is idle. That is, resources are reserved regardless of whether the queue is used. Dedicated queues ensure that resources exist when jobs are submitted.

Cross-AZ Queues

An AZ contains one or more physical data centers. Each AZ has independent cooling, fire extinguishing, moisture-proofing, and electricity facilities. Within an AZ, computing, network, storage, and other resources are logically divided into multiple clusters. AZs within a region are interconnected using high-speed optical fibers to allow you to build cross-AZ high-availability systems.

DLI dual-AZ queues improve data availability by creating a duplicate queue in the second AZ. You can continuously use DLI when one AZ is unavailable. The dual-AZ mode suits those who require high availability.

A DLI dual-AZ queue is created with the same compute resources in two AZs. For example, if you require 1,400 CUs, you can select 1,400 CUs and select the dual-AZ option when creating a queue. Then, DLI creates 1,400 CUs dedicated compute resources in two AZs. When one AZ is unavailable, the other AZ can properly process your compute tasks.

NOTE:

Currently, only SQL queues are supported.
The default queue and are not supported.
If you select Dual-AZ when creating a queue, the billing is twice as that in single-AZ mode.

AI-enhanced queue

AI-enhanced queues support SQL jobs specific to AI analysis. DLI uses AI images to integrate a series of common AI algorithm packages based on the original Spark images. Spark and AI capabilities can be combined, helping you run machine learning jobs. For details about the installation packages on which the AI image depends, see Dependency Packages for AI Image Installation. When creating a queue, select AI-enhanced for Queue Type.

Modifying Queue Specifications (Scaling)

After purchasing a queue, queue resources may be insufficient or wasted as the service scenario changes.

DLI allows you to scale in or out queue specifications. There are two adjustment modes: manual and scheduled adjustment.

Manual scale-out/scale-in: You can manually change the queue specifications based on service requirements. For details, see Modifying Queue Specifications (Manual Scale-out/Scale-in).
Scheduled scale-out/scale-in: You can set the time for automatic queue scaling based on service requirements. The system periodically triggers queue scaling. For details, see Scheduling CU Changes (Periodic Scale-out/Scale-in).

In addition, DLI can automatically trigger scale-in and scale-out for Flink jobs based on the job size.

NOTE:

Scaling can be performed for a newly created queue only when there are jobs running on this queue.

Queue Management Page

Queue Management provides the following functions:

NOTE:

To receive notifications when a DLI job fails, SMN Administrator permissions are required.

The queue list displays all queues created by you and the default queue. Queues are listed in chronological order by default in the queue list, with the most recently created queues displayed at the top.

**Table 2** Parameter description
Parameter	Description
Name	Name of a queue.
Type	Indicates the queue type. For SQL For general purpose Spark queue (compatible with earlier versions)
Queue Specifications	Queue size. Unit: CU CU is the pricing unit of queues. A CU consists of 1 vCPU and 4-GB memory. The computing capabilities of queues vary with queue specifications. The higher the specifications, the stronger the computing capability.
Owner	Queue owner.
Created	Time when the queue is created.
Description	Description of a queue specified during queue creation. If no description is provided, -- is displayed.
Operation	Details: Allow you to view the permission information of the queue and assign queue permissions to users. Restart: Forcibly restart a queue. NOTE: Only the SQL queue has the Restart operation. Delete: Allow you to delete the selected queue. You cannot delete a queue where there are running jobs or jobs are being submitted. Modifying CIDR Block: When DLI enhanced datasource connection is used, the CIDR block of the DLI queue cannot overlap with that of the data source. You can modify the CIDR block as required. Modify Queue Specifications: You can select Scale-out or Scale-in as required. The number of CUs after modification must be an integer multiple of 16. Create Periodic Task: You can set different queue sizes at different time or in different periods based on the service period or usage. The system automatically performs scale-out or scale-in as scheduled. The target value must be an integral multiple of 16 CUs. Manage Permissions: You can view the user permissions corresponding to the queue and grant permissions to other users. Test Address Connectivity: Test whether the queue is reachable to the specified address. The domain name and IP address are supported. The port can be specified.

Parent topic: Queue Management

Previous topic: Queue Management

Next topic: Queue Permission Management

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot