Example Use Case: Configuring Scaling Policies for Queues in an Elastic Resource Pool

Scenario

A company has multiple departments that perform data analysis in different periods during a day.

• Department A requires a large number of compute resources from 00:00 a.m. to 09:00 a.m. In other time segments, only small tasks are running.
• Department B requires a large number of compute resources from 10:00 a.m. to 10:00 p.m. Some periodical tasks are running in other time segments during a day.

In the preceding scenario, you can add two queues to an elastic resource pool: queue test_a for department A, and queue test_b for department B. You can add scaling policies for 00:00-09:00 and 10:00-23:00 respectively to the test_a and test_b queues. For jobs in other periods, you can modify the default scaling policy.

Precautions

• You are advised to implement fine-grained management of resource pools for stream and batch processing jobs by placing Flink real-time stream jobs and SQL batch processing jobs in separate elastic resource pools.

  Flink real-time stream jobs can run stably without forced scale-in, thus avoiding job interruption and system instability.

  SQL batch processing jobs are placed in independent resource pools, which can scale out and in more flexibly, significantly enhancing the success rate and operational efficiency of scaling operations.

• In any time segment of a day, the total minimum CUs of all queues in an elastic resource pool cannot be more than the minimum CUs of the pool.
• In any time segment of a day, the maximum CUs of any queue in an elastic resource pool cannot be more than the maximum CUs of the pool.
• The periods of scaling policies cannot overlap.
• The period of a scaling policy can only be set by hour and specified by the start time and end time. For example, if you set the period to 00-09, the period when the policy takes effect is [00:00, 09:00). The period of the default scaling policy cannot be modified.
• In any period, compute resources are preferentially allocated to meet the minimum number of CUs of all queues. The remaining CUs (maximum CUs of the elastic resource pool – total minimum CUs of all queues) are allocated in accordance with the scaling policy priorities.
  • Scaling policies with smaller priority values are prior to over those with larger priority values.
  • If the scaling policies of two queues have the same priority, resources are randomly allocated to a queue. If there are remaining resources, they are randomly allocated until there is no more left.

    Table 2 CU allocation

    Scenario	CUs
    An elastic resource pool has a maximum number of 256 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 128 Queue B: priority 10; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 160 CUs remaining. The remaining CUs are allocated based on the priorities. Queue B is prior to queue A. Therefore, queue B gets 64 CUs, and queue A has 96 CUs.
    An elastic resource pool has a maximum number of 96 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 64 Queue B: priority 10; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are no remaining CUs. The allocation is complete.
    An elastic resource pool has a maximum number of 128 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 64 Queue B: priority 10; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 32 CUs remaining. The remaining 32 CUs are all preferentially allocated to queue B.
    An elastic resource pool has a maximum number of 128 CUs for queue A and queue B. The scaling policies are as follows: Queue A: priority 5; period: 00:00–09:00; minimum CU: 32; maximum CU: 64 Queue B: priority 5; time period: 00:00–09:00; minimum CU: 64; maximum CU: 128	From 00:00 a.m. to 09:00 a.m.: The minimum CUs are allocated to the two queues. Queue A has 32 CUs, and queue B has 64 CUs. There are 32 CUs remaining. The two queues have the same priority, the remaining 32 CUs are randomly allocated to the two queues.

Setting a Scaling Policy

1. Log in to the DLI management console and create an elastic resource pool. Set the minimum and maximum number of CUs of the pool to 128 and 256 respectively. For details, see Creating an Elastic Resource Pool and Creating Queues Within It.
2. Choose Resources > Resource Pool. Locate the row that contains the created elastic resource pool, and click Queue MGMT in the Operation column.
3. Refer to Creating an Elastic Resource Pool and Creating Queues Within It to create the test_a queue and set the scaling policy.
   1. Set the priority of the default scaling policy to 5, Min CU to 16, and Max CU to 32.
   2. Click create to add a scaling policy. Set the priority to 20, Period to 00--09, Min CU to 64, and Max CU to 128.
   Figure 1 Adding a scaling policy for queue test_a
   

4. View the scaling policy on the Queue Management page of the specific elastic resource pool.
   Figure 2 Viewing the new scaling policy
   
   Click to view graphical statistics of priorities and CU settings for all time segments.

   Figure 3 Scaling policies
   

5. Refer to Creating an Elastic Resource Pool and Creating Queues Within It to create the test_b queue and set the scaling policy.
   1. Set the priority of the default scaling policy to 5, Min CU to 32, and Max CU to 64.
   2. Click create to add a scaling policy. Set the priority to 20, Period to 10--23, Min CU to 64, and Max CU to 128.
   Figure 4 Adding a scaling policy for queue test_b
   

6. View the scaling policy on the Queue Management page of the specific elastic resource pool.
   Figure 5 Viewing the new scaling policy
   

   Click to view graphical statistics on priorities and CU settings of the two queues for all time segments.

   Figure 6 Scaling policies of both queues