Example Use Case: Configuring Scaling Policies for Queues in an Elastic Resource Pool

Updated on 2024-12-28 GMT+08:00

View PDF

Scenario

A company has multiple departments that perform data analysis in different periods during a day.

Department A requires a large number of compute resources from 00:00 a.m. to 09:00 a.m. In other time segments, only small tasks are running.
Department B requires a large number of compute resources from 10:00 a.m. to 10:00 p.m. Some periodical tasks are running in other time segments during a day.

In the preceding scenario, you can add two queues to an elastic resource pool: queue test_a for department A, and queue test_b for department B. You can add scaling policies for 00:00-09:00 and 10:00-23:00 respectively to the test_a and test_b queues. For jobs in other periods, you can modify the default scaling policy.

**Table 1** Scaling policy
Queue	Period	Priority	CUs	Default Period	Default Priority	Default CUs	Remarks
test_a	[00:00, 09:00)	20	Minimum CU: 64 Maximum CU: 128	The time segments beyond [00:00, 09:00)	5	Minimum CU: 16 Maximum CU: 32	Jobs of department A
test_b	[10:00, 23:00)	20	Minimum CU: 64 Maximum CU: 128	The time segments beyond [10:00, 23:00)	5	Minimum CU: 32 Maximum CU: 64	Jobs of department B

Precautions

You are advised to implement fine-grained management of resource pools for stream and batch processing jobs by placing Flink real-time stream jobs and SQL batch processing jobs in separate elastic resource pools.
Flink real-time stream jobs can run stably without forced scale-in, thus avoiding job interruption and system instability.

SQL batch processing jobs are placed in independent resource pools, which can scale out and in more flexibly, significantly enhancing the success rate and operational efficiency of scaling operations.
In any time segment of a day, the total minimum CUs of all queues in an elastic resource pool cannot be more than the minimum CUs of the pool.
In any time segment of a day, the maximum CUs of any queue in an elastic resource pool cannot be more than the maximum CUs of the pool.
The periods of scaling policies cannot overlap.
The period of a scaling policy can only be set by hour and specified by the start time and end time. For example, if you set the period to 00-09, the period when the policy takes effect is [00:00, 09:00). The period of the default scaling policy cannot be modified.

In any period, compute resources are preferentially allocated to meet the minimum number of CUs of all queues. The remaining CUs (maximum CUs of the elastic resource pool – total minimum CUs of all queues) are allocated in accordance with the scaling policy priorities.

Scaling policies with smaller priority values are prior to over those with larger priority values.

If the scaling policies of two queues have the same priority, resources are randomly allocated to a queue. If there are remaining resources, they are randomly allocated until there is no more left.

**Table 2** CU allocation
Scenario	CUs
An elastic resource pool has a maximum number of 256 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 128 Queue B: priority 10; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 160 CUs remaining. The remaining CUs are allocated based on the priorities. Queue B is prior to queue A. Therefore, queue B gets 64 CUs, and queue A has 96 CUs.
An elastic resource pool has a maximum number of 96 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 64 Queue B: priority 10; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are no remaining CUs. The allocation is complete.
An elastic resource pool has a maximum number of 128 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 64 Queue B: priority 10; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 32 CUs remaining. The remaining 32 CUs are all preferentially allocated to queue B.
An elastic resource pool has a maximum number of 128 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 64 Queue B: priority 5; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 32 CUs remaining. The two queues have the same priority, the remaining 32 CUs are randomly allocated to the two queues.

Setting a Scaling Policy

Log in to the DLI management console and create an elastic resource pool. Set the minimum and maximum number of CUs of the pool to 128 and 256 respectively. For details, see Creating an Elastic Resource Pool and Creating Queues Within It.
Choose Resources > Resource Pool. Locate the row that contains the created elastic resource pool, and click Queue MGMT in the Operation column.
Refer to Creating an Elastic Resource Pool and Creating Queues Within It to create the test_a queue and set the scaling policy.
1. Set the priority of the default scaling policy to 5, Min CU to 16, and Max CU to 32.
2. Click create to add a scaling policy. Set the priority to 20, Period to 00--09, Min CU to 64, and Max CU to 128.
Figure 1 Adding a scaling policy for queue test_a
View the scaling policy on the Queue Management page of the specific elastic resource pool.

Figure 2 Viewing the new scaling policy

Click to view graphical statistics of priorities and CU settings for all time segments.
Figure 3 Scaling policies
Refer to Creating an Elastic Resource Pool and Creating Queues Within It to create the test_b queue and set the scaling policy.
1. Set the priority of the default scaling policy to 5, Min CU to 32, and Max CU to 64.
2. Click create to add a scaling policy. Set the priority to 20, Period to 10--23, Min CU to 64, and Max CU to 128.
Figure 4 Adding a scaling policy for queue test_b
View the scaling policy on the Queue Management page of the specific elastic resource pool.

Figure 5 Viewing the new scaling policy

Click to view graphical statistics on priorities and CU settings of the two queues for all time segments.

Figure 6 Scaling policies of both queues

Parent topic: Creating an Elastic Resource Pool and Queues Within It

Previous topic: Example Use Case: Creating an Elastic Resource Pool and Running Jobs

Next topic: Creating a Non-Elastic Resource Pool Queue (Discarded and Not Recommended)

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot