What's New

1

Added APIs related to resource tags

APIs related to resource tags are added to DLI. These APIs can be used to add and query resource tags.

Commercial use

Batch Adding Resource Tags

1

Optimized the content related to DLI agencies

The description of DLI agencies, including their types and use cases, is optimized.

Commercial use

DLI Agency Overview

2

Modified the method for creating compute resources in "Getting Started" in the Data Lake Insight API Reference

The method for creating compute resources is modified in "Getting Started" in the Data Lake Insight API Reference. Specifically, the procedure for creating an elastic resource pool and adding queues within it is added.

Commercial use

Creating and Submitting a SQL Job Using an API

1

Added "DLI Job Development Process" to the Data Lake Insight User Guide

The "DLI Job Development Process" section that describes operations of different types of DLI jobs is added to the Data Lake Insight User Guide.

Commercial use

DLI Job Development Process

1

Optimized the Data Lake Insight User Guide

• The DLI job development process is added.

• Preparations for using DLI are added.

• Instructions on how to migrate external data sources to DLI are added, along with migration cases in typical scenarios.

• The section "DLI Cloud Data" is added, which describes basic concepts of tables, data sources, data catalogs, and metadata.

• The structure of the manual for DLI to read and write data from and to external data sources is optimized, and instructions on how to use DEW to manage access credentials for data sources are added.

• The structure of the usage guide for different types of jobs is optimized.

• Instructions on how to use basic datasource connections and create Flink SQL jobs are taken offline.

Commercial use

Overview of Data Migration Scenarios

Understanding Tables

Databases

and Data Catalogs

Using DEW to Manage Access Credentials for Data Sources

1

Optimized the Data Lake Insight What's New

It is optimized as follows:

• It is adapted to the new version of the DLI engine.

• It outlines the steps for submitting jobs to queues created within a DLI elastic resource pool.

Commercial use

Using DLI to Submit a SQL Job to Query OBS Data

Using DLI to Submit a SQL Job to Query RDS for MySQL Data

Using DLI to Submit a Flink OpenSource SQL Job to Query RDS for MySQL Data

1

Support for DLI jobs to access resources in a shared VPC

DLI now allows for establishing network connections between DLI and resources in a shared VPC, enabling access to those resources when submitting jobs.

Commercial use

Establishing a Network Connection Between DLI and Resources in a Shared VPC

2

Added audit events related to Flink OpenSource SQL jobs

Audit events related to Flink OpenSource SQL jobs are added.

Commercial use

DLI Operations That Can Be Recorded by CTS

1

Added support for Spark 3.3.1

DLI follows the release consistency of the open source Spark compute engine and now supports version 3.3.1.

Commercial use

Spark Syntax Reference

2

Added support for Flink 1.15

DLI follows the release consistency of the open source Flink compute engine and now supports version 1.15.

Commercial use

Flink OpenSource SQL 1.15 Syntax Reference

3

Support for visual configuration of SQL inspection rules

DLI supports visual configuration of SQL inspection rules, providing proactive defense against typical large and low-quality SQL queries, including hint, interception, and blocking.

Commercial use

Configuring SQL Inspection Rules

4

Agency permission minimization

DLI upgrades the system agency from dli_admin_agency to dli_management_agency. The new agency includes the necessary permissions for obtaining IAM user information, datasource operations, and message notifications, effectively avoiding uncontrolled authorization issues with DLI-related services.

Commercial use

Updating Agency Permissions

1

Added constraints on tag policy usage

DLI has added constraints on the usage of tag policies, which means that tags are added to resources according to the tag policy rules set by the organization.

Commercial use

Tag Management

1

Added support documentation for Spark open source commands

DLI has added support documentation for Spark open source commands.

Commercial use

Spark Open Source Commands

1

Added Spark 3.1.1 dependencies

DLI has added information on Spark 3.1.1 dependencies.

Commercial use

Built-in Dependencies

2

Added data authorization API

DLI has added a data authorization API that allows for the assignment of database or table data permissions to specified users or projects.

Commercial use

Granting Users with the Data Usage Permission

3

Added monitoring metrics related to elastic resource pools

DLI has added monitoring metrics related to elastic resource pools.

Commercial use

DLI Monitoring Metrics

1

Optimized the examples in Data Lake Insight Spark SQL Syntax Reference

More examples for creating OBS and DLI tables are added to the document.

Commercial use

Creating an OBS Table Using the DataSource Syntax

Creating an OBS Table Using the Hive Syntax

Creating a DLI Table Using the DataSource Syntax

Creating a DLI Table Using the Hive Syntax

2

Added Data Lake Insight Billing

The document covers the DLI billed items, billing description, and billing examples.

Commercial use

Data Lake Insight Billing

1

Added support for viewing Total CPU Used and Output Bytes of SQL jobs

You can view Total CPU Used and Output Bytes of a SQL job whose Type is IMPORT or QUERY.

Commercial use

SQL Job Management

2

Added the description of the .sha256 file in the SDK installation package

Added the description of downloading the .sha256 file corresponding to the SDK installation package to Prerequisites.

Commercial use

Preparing the SDK Environment

1

Permission Management for Global Variables

Added Permission Management for Global Variables.

Commercial use

Permission Management for Global Variables

2

Added support for allocating queues to projects

You can allocate queues to projects.

Commercial use

Allocating a Queue to an Enterprise Project

1

Added section "Constraints" to Data Lake Insight Service Overview

Constraints and limitations on using DLI are added.

Commercial use

Constraints

2

Elastic Resource Pool

Describes basic operations on elastic resource pools.

Open beta testing

Elastic Resource Pool

1

Spark 3.1.1 images

Adds the address for downloading Spark 3.1.1 images.

Commercial use

Spark 3.1.1 images

2

Built-in Functions

Adds mathematical, date, string, and other functions to Spark built-in functions.

Commercial use

Built-in Functions

3

Datasource Authentication

Describes the operations related to datasource authentication.

Commercial use

Datasource Authentication

4

Enhanced Datasource Connections

Describes the operations related to enhanced datasource connections.

Commercial use

Enhanced Datasource Connections

1

Dynamic scaling for Flink jobs

Enables dynamic scaling for Flink jobs.

Commercial use

Dynamic scaling for Flink jobs

2

Job priority

Sets the priority of a job.

Commercial use

Job priority

1

DLI queue tags

Modifies the description of the queue tag key and tag value.

Commercial use

DLI queue tags

1

New datasource connection APIs

Call APIs to create, list, obtain, update, and delete datasource authentication, and create and delete routes.

Commercial use

Datasource Authentication APIs

Creating a Route

Creating a Route

2

Template-related APIs

APIs for creating job templates and updating SQL templates are now available.

Commercial use

Templates

1

SNAT rules for connecting DLI queues to the public network

Enable communications between a queue and the Internet by configuring SNAT rules and adding routes to the public network.

Commercial use

Configuring the Connection Between a DLI Queue and a Data Source in the Internet

1

New section

Use Flink Jar to connect to a Kafka with SASL_SSL authentication enabled.

Commercial use

Using Flink Jar to Connect to a Kafka with SASL_SSL Authentication Enabled

1

Flink OpenSource SQL 1.12 syntax

DLI now supports jobs that use Flink OpenSource SQL 1.12 syntax.

Commercial use

Flink OpenSource SQL 1.12 Syntax

1

Best practices for using CDM to migrate data to DLI

You can use CDM to easily migrate data from other cloud services or service platforms to DLI.

Commercial use

Migrating Data from Hive to DLI

1

Guide for using Spark Jar jobs to read and query OBS data

You can find helpful guide on how to write a Spark program to read and query OBS data, compile and package your code, and submit a Spark Jar job.

Commercial use

Using Spark Jar Jobs to Read and Query OBS Data

2

Guide for using UDTFs in Spark SQL jobs

DLI allows you to use Hive user-defined table-generating functions (UDTFs) to customize table-valued functions. Hive UDTFs are widely used in one-in-multiple-out queries and analysis.

Commercial use

Calling UDTFs in Spark SQL Jobs

1

Flink OpenSource SQL syntax for ClickHouse and user-defined tables

You can refer to the syntax description for using user-defined source table and result table and ClickHouse result table in Flink OpenSource SQL jobs.

Commercial use

User-defined Source Table

User-defined Result Table

ClickHouse Result Table

1

Multiple data versions

DLI controls multiple versions of backup data for restoration.

Commercial use

Enabling or Disabling Multiversion Backup

2

Flink 1.11

DLI now supports Flink 1.11.

--

Apache Flink Documentation

1

Custom images

DLI supports clusters deployed in containers. In a container cluster, components related to Spark jobs and Flink jobs run in containers. You can download custom images provided by DLI to change the container running environments of Spark jobs and Flink jobs.

Commercial use

Overview of Custom Images

1

Package billed by storage

DLI provides storage packages for you to reduce the cost of storing data in DLI.

Commercial use

Billing

1

Dual-AZ compute queues

Cross-AZ queues can quickly recover from disasters, improving computing reliability.

Commercial use

Queue Management Overview

2

Spark job developer mode

You can set parameters by calling APIs on the DLI management console.

Commercial use

Creating a Batch Processing Job

1

Data scanning package

You can buy a scanning package to reduce the fee charged for the amount of data scanned by each job.

Commercial use

Billing

2

Global variables configuration

DLI allows you to set global variables to protect key user information.

Commercial use

Global Variables

1

IAM fine-grained authorization

You can mange fine-grained permissions of DLI with IAM.

Commercial use

Creating a Custom Policy

2

Flink 1.10

DLI now supports Flink 1.10.

Commercial use

Apache Flink Documentation