Compute Resource Types and Specifications

Video Introduction

This video introduces the compute resource types of DLI.

What Are Elastic Resource Pools and Queues?

Before we dive into the compute resource modes of DLI, let us first understand the basic concepts of elastic resource pools and queues.

• Elastic resource pool

  An elastic resource pool is a pooled management mode for DLI compute resources, which can be seen as a collection of DLI compute resources. DLI supports the creation of multiple queues within an elastic resource pool, and these queues can share the resources in the elastic resource pool.

  For details about the specifications of elastic resource pools, see Elastic Resource Pool Specifications.

  Figure 1 shows the architecture of an elastic resource pool. For more information about the advantages of elastic resource pools, see Overview of DLI Elastic Resource Pools and Queues.

  • The physical resource layer of an elastic resource pool consists of compute nodes distributed in different AZs.
  • Multiple queues within the same resource pool share physical resources but maintain logical isolation to enforce resource allocation policies such as priorities and quotas.
  • Elastic resource pools can adjust resources in real time based on queue loads, achieving on-demand auto scaling in minutes.
  • An elastic resource pool can simultaneously support SQL, Spark, and Flink jobs. The specific job types supported depend on the queue types created within the elastic resource pool.
    Refer to DLI Compute Resource Modes and Supported Queue Types.

    Figure 1 Elastic resource pool architecture
    
• Queue

  Queues are the basic units of compute resources that are actually used and allocated in DLI. You can create different queues for different jobs or data processing tasks, and allocate and adjust resources for these queues as needed.

  DLI is divided into three queue types: default queue, for SQL, and for general purpose. You can choose the most suitable queue type based on your business scenario and job characteristics.

  • default queue:

    The default queue is a preset queue that is shared among all users.

    The default queue does not support specifying the size of resources and resources are allocated on-demand during job execution, with billing based on the actual amount of data scanned.

    As resources of the default queue are shared among all users, there may be resource contention during use, and it cannot be guaranteed that resources will be available for every job execution.

    The default queue is suitable for small-scale or temporary data processing needs. For important jobs or jobs that require guaranteed resources, you are advised to buy an elastic resource pool and create queues within it to execute jobs.

  • For SQL:

    For SQL queues are used to execute SQL jobs and supports specifying engine types including Spark and HetuEngine.

    This type of queues is suitable for businesses that require fast data query and analysis, as well as regular cache clearing or environment resetting.

  • For general purpose:

    For general purpose queues are used to execute Spark jobs, Flink OpenSource SQL jobs, and Flink Jar jobs.

    It is suitable for complex data processing, real-time data stream processing, or batch data processing scenarios.

DLI Compute Resource Modes

DLI offers three compute resource management modes, each with unique advantages and use cases.

Figure 2 DLI compute resource modes

• Elastic resource pool mode:
  A pooled management mode for compute resources that provides dynamic scaling capabilities. Queues within the same elastic resource pool share compute resources. By appropriately setting up compute resource allocation policies for queues, you can enhance compute resource utilization and handle peak business demands efficiently.

  • Use cases: suitable for scenarios with significant fluctuations in business volume, such as periodic data batch processing tasks or real-time data processing needs.
  • Supported queue types: for SQL (Spark), for SQL (HetuEngine), and for general purpose. For details about DLI queue types, see Queue Types.
    

    General-purpose queues and SQL queues in elastic resource pool mode do not support cross-AZ deployment.

  • Usage: first create an elastic resource pool, then create queues within the pool and allocate compute resources. Queues are associated with specific jobs and data processing tasks.

    For details about how to purchase an elastic resource pool and create queues within the pool, see Creating an Elastic Resource Pool and Creating Queues Within It.

• Global sharing mode:

  Global sharing mode is a compute resource allocation mode that allocates resources based on the actual amount of data scanned in SQL queries. It does not support specifying or reserving compute resources.

  The default queue, which is preset by DLI, is the compute resource for global sharing mode, and the resource size is allocated on demand. Users who are unsure of the data size or occasionally need to process data can use the default queue to run jobs.

  • Use cases: suitable for testing jobs or scenarios with low resource consumption.
  • Supported queue types: Only the preset default queue in DLI is the compute resource for global sharing mode.

    The default queue is typically used by users who are new to DLI but it may lead to resource contention and prevent you from getting the resources you need for your jobs, as its resources are shared among all users. You are advised to use self-built queues to run production jobs.

  • Usage: The default queue is only applicable to submitting SQL jobs. When submitting SQL jobs on the DLI management console, select the default queue.
• Non-elastic resource pool mode (deprecated and not recommended):

  The previous-gen of DLI's compute resource management mode is no longer recommended due to its lack of flexibility.

  Non-elastic resource pool mode provides fixed-specification compute resources that are purchased and exclusively used, and cannot be dynamically adjusted according to demand, which may result in resource waste or insufficient resources during peak hours.

To help you understand the use cases for different DLI compute resource modes, we can compare purchasing DLI compute resources to using car services:

• The elastic resource pool mode can be compared to "renting a car" where you can dynamically adjust the scale of resources based on actual needs.

  This mode is suitable for scenarios with significant fluctuations in business demand, allowing for flexible adjustment of resources based on peak and off-peak demands to optimize costs.

• The global sharing mode can be compared to "taking a taxi" where you only pay for the actual amount of data used.

  This mode is suitable for scenarios where data size is uncertain or data processing is only required occasionally, allowing for on-demand use of resources without the need to pre-purchase or reserve resources.

DLI Compute Resource Modes and Supported Queue Types

Table 1 describes the queue types supported by different compute resources in DLI.

Elastic Resource Pool Specifications

An elastic resource pool provides compute resources (CPU and memory) for running DLI jobs. The unit is CU. One CU contains 1 vCPU and 4 GB memory.

You can create multiple queues in an elastic resource pool. Compute resources can be shared among queues. By appropriately setting up compute resource allocation policies for queues, you can enhance compute resource utilization.

DLI elastic resource pools are physically isolated, while queues within the same elastic resource pool are logically isolated.

You are advised to create separate elastic resource pools for testing and production scenarios to ensure the independence and security of resource management through physical isolation.

Table 2 lists the specifications of the elastic resource pools provided by DLI.