Function Overview

DLI has a comprehensive permission control mechanism and supports fine-grained authentication through Identity and Access Management (IAM). You can create policies in IAM to manage DLI permissions. You can use both the DLI's permission control mechanism and the IAM service for permission management.

The DLI provides the elastic queue scaling function. Users can adjust queue specifications based on their service usage or usage period to meet service requirements and reduce costs.

An availability zone (AZ) contains one or multiple physical data centers. Each AZ has independent cooling, fire extinguishing, moisture-proof, and electricity facilities. Within an AZ, computing, network, storage, and other resources are logically divided into multiple clusters. AZs within a region are interconnected using high-speed optical fibers to support cross-AZ high-availability systems. 

DLI cross-AZ queues provide cross-region DR capabilities for users, improving computing reliability. Users can continuously use DLI services when a single AZ is unavailable.This mode applies to scenarios where users have high requirements on queue reliability.
DLI cross-AZ queues are used to create the same computing resources in two different AZs. For example, if user 1 requires 1400 CUs computing resources, user 1 can select 1400 CUs computing resources and select Cross-AZ when creating a queue. DLI creates dedicated 1400 CUs computing resources in two different AZs for user 1. If one AZ is unavailable, the other AZ can properly process user 1's computing tasks.

Description:
Currently, only SQL queues are supported.
Currently, only yearly/monthly queues and on-demand dedicated queues can be used for cross-AZ active-active. Common on-demand queues and default queues are not supported.
If cross-AZ is selected during queue purchase, the cross-AZ billing rate is twice that in single-AZ mode.

DLI SQL jobs are DLI Spark SQL jobs. You can use SQL statements in the SQL editor to perform operations such as data query.Supports SQL2003 and is compatible with SparkSQL. For details about the syntax, see the Data Lake Exploration Spark SQL Syntax Reference.

Based on the open-source Spark, DLI performs a lot of performance optimization and service-oriented reconstruction, is compatible with the Apache Spark ecosystem and interfaces, and executes batch processing tasks.
DLI can also use Spark jobs to access DLI metadata. For details, see the Data Lake Insight Development Guide.

DLI Flink jobs support online Flink SQL analysis and cross-source connection with multiple cloud services to form a rich stream ecosystem.
Currently, the following Flink job types are available:

• Flink Jar job: a customized JAR package job based on the Flink API. It can run on an exclusive queue.

Before using DLI to perform cross-source analysis, you need to establish a datasource connection to establish the network between data sources.
The DLI enhanced datasource connection uses a peer-to-peer connection at the bottom layer to directly connect the VPC network between the DLI queue and the destination data source. The DLI enhanced datasource connection implements point-to-point data communication, providing more flexible application scenarios and stronger performance than the typical cross-source connection.
Description:
The default queue does not support the creation of datasource connections.

The enhanced cross-source function supports all cross-source services implemented by DLI and allows users to access self-built data sources through UDF, Spark, and Flink jobs.The enhanced cross-source scenario supports only yearly/monthly queues and on-demand dedicated queues.
Currently, DLI supports cross-source access to the following data sources: CloudTable HBase, CloudTable OpenTSDB, CSS, DCS Redis, DDS Mongo, DIS, DMS, DWS, MRS HBase, MRS Kafka, MRS OpenTSDB, and OBS. RDS MySQL, RDS PostGre, and SMN.For details, see Data Sources Supported by DLI.
Description:
To access a cross-source table, you need to use the queue for which a cross-source connection has been created.
Cross-source tables do not support the preview function.

DLI supports clusters deployed in containers. In a container cluster, components related to Spark jobs and Flink jobs run in containers. You can download custom images provided by DLI to change the container running environments of Spark jobs and Flink jobs. For example, you can add a Python package or C library related to machine learning to a custom image to facilitate function extension.

Usage Restrictions

• The DLI container queue must be used and it must be a dedicated queue.
• The basic image provided by DLI must be used.
• You cannot modify the DLI components and directories in the basic image.
• Only JAR jobs (Spark and Flink Jar jobs) are supported.