Basic Concepts

• Actual CUs dynamically scale within the CU range defined by [Minimum CUs, Maximum CUs] based on the queue load.
• If actual CUs ≤ yearly/monthly CUs, billing is entirely based on the yearly/monthly rate with no additional pay-per-use charges.
• If actual CUs > yearly/monthly CUs, the difference is billed hourly at the pay-per-use rate.

Figure 1 Relationship between the actual CUs, used CUs, CU range, and yearly/monthly CUs of DLI

Constraints for setting elastic resource pool CUs:

• The minimum CUs (minCU) of an elastic resource pool must be less than or equal to the actual CUs. If expanding minCU exceeds the current actual CUs, you must first increase the actual CUs. Otherwise, the modification will fail.
• The sum of all queues' minimum CUs in an elastic resource pool must not exceed the pool's minCU.
• Any single queue's maxCU cannot exceed the pool's maxCU.
• Adjustments to a queue's CU range, changes to the pool's yearly/monthly CUs, or modifications to the pool's CU settings take effect at the next full hour.
• Increasing the number of queues to adjust the pool's actual CUs takes immediate effect.

• Actual CUs are the total amount of compute resources currently allocated in an elastic resource pool for running jobs (measured in CUs). They also serve as the direct basis for billing.
• Calculation rules for actual CUs:
  • When no queues exist in the resource pool: The actual CUs equal the minimum CUs of the elastic resource pool.
  • When there are queues in the resource pool, the formula to calculate actual CUs is:
    • Actual CUs = max{(min[sum(maximum CUs of queues), maximum CUs of the elastic resource pool]), minimum CUs of the elastic resource pool}.

      

    • The result must be a multiple of 16 CUs. If not divisible by 16, round up to the nearest multiple.
• Billing modes:

  Table 1 Billing modes

  Billing Mode	Billing Object	Description
  Pay-per-use	Based on actual CUs	For details, see Billing Modes.
  Yearly/Monthly	Divided into two parts: within specifications and beyond specifications	Within specifications: billed on a yearly/monthly basis. Beyond specifications: excess CUs (actual CUs – specifications) are billed on a pay-per-use basis. Cost optimization tip: To achieve more favorable pricing, adjust the elastic resource pool's specifications to match the actual CUs. This ensures all CUs are billed on a yearly/monthly basis, resulting in overall cost savings.

• Relationship between actual CUs and elastic resource pool scaling

  Scaling out or in an elastic resource pool means adjusting its actual CUs.

  Refer to Scaling Out or In an Elastic Resource Pool and Actual CUs Calculation Formula.

• Example of actual CU allocation:

  Consider Table 2 below, which illustrates the process of calculating actual CUs for an elastic resource pool:

  1. Calculate the sum of maximum CUs of the queues: sum(maximum CUs) = 32 + 56 = 88 CUs.
  2. Compare the sum of maximum CUs of the queues with the maximum CUs of the elastic resource pool and take the smaller value: min{88 CUs, 112 CUs} = 88 CUs.
  3. Compare the value with the minimum CUs of the elastic resource pool and take the larger value: max(88 CUs, 64 CUs) = 88 CUs.
  4. Check if 88 CUs is a multiple of 16 CUs. Since 88 is not divisible by 16, round up to 96 CUs.

     Table 2 Example of actual CU allocation of an elastic resource pool

     Scenario	Resource Type	CU Range
     New elastic resource pool: 64–112 CUs Queues A and B are created within the elastic resource pool. The CU ranges of the two queues are: CU range of queue A: 16–32 CUs CU range of queue B: 16–56 CUs	Elastic resource pool	64–112 CUs
     	Queue A	16–32 CUs
     	Queue B	16–56 CUs

• Definition:

  Used CUs are the number of CUs actively occupied by running jobs or tasks at a given moment. There is a 5-minute delay in reporting used CUs.

• Difference from actual CUs:

  Actual CUs refer to the total number of CUs allocated to the resource pool, whereas used CUs indicate the portion currently used for task execution.

• Notes and constraints
  • In HetuEngine scenarios, used CUs are equal to actual CUs.
  • For other engines like Flink and Spark, actual CUs may exceed used CUs, with the difference representing idle but immediately available resources.

Figure 2 Used CUs

The CU range acts as a safety fence for scaling elastic resource pools, defined by two boundaries: minimum CUs (minCU) and maximum CUs (maxCU). The actual CUs of the elastic resource pool automatically scale within this range, preventing unlimited expansion risks.

When calculating actual CUs: If the result is below minCU, actual CUs default to minCU. If the result exceeds maxCU, actual CUs cap at maxCU. Thus, actual CUs always remain within the CU range.

• Minimum CUs (minCU)

  In yearly/monthly mode, the minCU set during purchase is the yearly/monthly CUs (specifications) for the elastic resource pool. This can be increased on the Modify Yearly/Monthly CU page.

  During resource pool expansion, the minCU is linked to and changes accordingly to match the yearly/monthly CUs (specifications), while maxCU remains unchanged.

  During resource pool scale-in, actual CUs never fall below minCU, ensuring immediate availability for job takeover.

  The sum of all queues' minimum CUs in an elastic resource pool must not exceed the pool's minCU.

• Maximum CUs (maxCU)

  Any single queue's maxCU cannot exceed the pool's maxCU.

  The resource pool ensures it meets the minCU requirements across all queues while striving to accommodate their maxCU demands.

This refers to the CU quantity purchased under the yearly/monthly mode. At purchase, the minCU of the CU range serves as the yearly/monthly CUs (specifications).

The chosen minimum CU value during purchase defines the elastic resource pool's specifications, which is unique to yearly/monthly resource pools. Resources within the specifications are billed on a yearly/monthly basis, and excess usage incurs pay-per-use charges.

The queue's maximum CUs directly impact actual CU allocation. If actual CUs exceed the specifications, the excess portion is billed at the pay-per-use rate. To optimize costs in such scenarios, you can increase the elastic resource pool's yearly/monthly CUs (specifications). Once updated, all resources within the new yearly/monthly CUs (specifications) will be billed under the more cost-effective yearly/monthly mode, eliminating partial pay-per-use charges.

Figure 3 Elastic resource pool – yearly/monthly + pay-per-use billing | cost optimization example

A queue is the fundamental unit for allocating and using compute resources in DLI to execute jobs. You can create separate queues for different jobs or data processing tasks and allocate or adjust their resources as needed.

Elastic resource pools in DLI are physically isolated clusters, while queues within the same pool are logically isolated.

For better resource management and security, you are advised to create distinct elastic resource pools for testing and production environments.

DLI offers three queue types: default, for SQL, and for general purpose. Choose the most suitable type based on your business needs and job characteristics.

• default queue:

  Pre-configured by DLI and shared among all users.

  Resources cannot be preallocated. They are assigned dynamically during job execution and billed based on the amount of data scanned.

  As a shared resource, there may be contention, and access to resources is not guaranteed.

  It is ideal for small-scale or temporary data processing tasks. For critical workloads requiring assured resources, consider purchasing an elastic resource pool and creating dedicated queues.

• For SQL:

  Designed specifically for SQL jobs, supporting engines such as Spark and HetuEngine.

  Suitable for fast data queries, analytics, periodic cache clearing, or environment resets.

• For general purpose:

  Used for executing Spark jobs, Flink OpenSource SQL jobs, and Flink Jar jobs.

  Best suited for complex data processing, real-time stream processing, or batch data processing scenarios.

Actual CUs of a queue are the total amount of compute resources allocated to that specific queue at a given moment, which are immediately available for use.

These actual CUs dynamically scale within the CU range defined by [Minimum CUs, Maximum CUs] based on the queue load.

• Definition:

  Used CUs of a queue are the number of CUs currently occupied by running jobs or tasks at a given moment. There is a 5-minute delay in reporting used CUs.

• Difference from actual CUs:

  Actual CUs refer to the total number of CUs allocated to the queue, whereas used CUs indicate the portion currently used for task execution.

Minimum and maximum CUs of a queue define its scaling policies, set during creation. These limits act as safety barriers, ensuring the queue's actual CUs fluctuate only within this predefined range and preventing unlimited scaling.

The maximum CU setting directly impacts the allocation of actual CUs in the elastic resource pool. If the maximum CU value is too high, it may cause the elastic resource pool's actual CUs to exceed its yearly/monthly CUs (specifications), resulting in additional pay-per-use charges for the excess usage.

To optimize costs in such scenarios, you can increase the elastic resource pool's yearly/monthly CUs (specifications).

Once updated, all resources within the new yearly/monthly CUs (specifications) will be billed under the more cost-effective yearly/monthly mode, eliminating partial pay-per-use charges.