What's New

1

Optimized the Data Lake Insight What's New

It is optimized as follows:

• It is adapted to the new version of the DLI engine.
• It outlines the steps for submitting jobs to queues created within a DLI elastic resource pool.

Commercial use

Using DLI to Submit a SQL Job to Query OBS Data

Using DLI to Submit a SQL Job to Query RDS for MySQL Data

Using DLI to Submit a Flink OpenSource SQL Job to Query RDS for MySQL Data

1

Commercial use

DLI Operations That Can Be Recorded by CTS

1

Added support for Spark 3.3.1

DLI follows the release consistency of the open source Spark compute engine and now supports version 3.3.1.

Commercial use

Spark Syntax Reference

2

Added support for Flink 1.15

DLI follows the release consistency of the open source Flink compute engine and now supports version 1.15.

Commercial use

Flink OpenSource SQL 1.15 Syntax Reference

3

Support for visual configuration of SQL inspection rules

DLI supports visual configuration of SQL inspection rules, providing proactive defense against typical large and low-quality SQL queries, including hint, interception, and blocking.

Commercial use

Configuring SQL Inspection Rules

4

Agency permission minimization

DLI upgrades the system agency from dli_admin_agency to dli_management_agency. The new agency includes the necessary permissions for obtaining IAM user information, datasource operations, and message notifications, effectively avoiding uncontrolled authorization issues with DLI-related services.

Commercial use

Updating Agency Permissions

1

Added notes and constraints on tag policy usage

Notes and constraints on the use of DLI tag policies are added, which means that tags are added to resources according to the tag policy rules set by the organization.

Commercial use

Tag Management

1

Added support documentation for Spark open source commands

Support documentation is added for DLI Spark open source commands.

Commercial use

Spark Open Source Commands

1

Added Spark 3.1.1 dependencies

Information on DLI Spark 3.1.1 dependencies is added.

Commercial use

Built-in Dependencies

2

Added data authorization API

A data authorization API that allows for the assignment of database or table data permissions to specified users or projects has been added.

Commercial use

Granting Users with the Data Usage Permission

3

Added monitoring metrics related to elastic resource pools

Monitoring metrics related to DLI elastic resource pools are added.

Commercial use

DLI Monitoring Metrics

1

Added Data Lake Insight Billing.

The document covers the DLI billed items, billing description, and billing examples.

Commercial use

Data Lake Insight Billing

2

Optimized the examples in Data Lake Insight Spark SQL Syntax Reference.

More examples for creating OBS and DLI tables are added to the document.

Commercial use

Creating an OBS Table Using the DataSource Syntax

Creating an OBS Table Using the Hive Syntax

Creating a DLI Table Using the DataSource Syntax

Creating a DLI Table Using the Hive Syntax

1

Added support for viewing Total CPU Used and Output Bytes of SQL jobs.

You can view Total CPU Used and Output Bytes of a SQL job whose Type is IMPORT or QUERY.

Commercial use

SQL Job Management

2

Added the description of the .sha256 file in the SDK installation package.

You can download the .sha256 file corresponding to the SDK installation package as instructed.

Commercial use

Prerequisites

1

Added support for allocating queues to projects.

You can allocate queues to projects.

Commercial use

Allocating a Queue to an Enterprise Project

2

Added the function of managing permissions for global variables.

You can manage permissions for global variables.

Commercial use

Permission Management for Global Variables

1

Added section "Notes and Constraints" to Data Lake Insight Service Overview.

Notes and constraints on using DLI are added.

OBT

Notes and Constraints

1

Added the address for downloading Spark 3.1.1 images.

You can download Spark 3.1.1 images through the address provided.

Commercial use

Spark 3.1.1 Image

2

Enhanced datasource connections

Describes the operations related to enhanced datasource connections.

Commercial use

Enhanced Datasource Connections

3

Datasource authentication

Describes the operations related to datasource authentication.

Commercial use

Datasource Authentication

4

Spark built-in functions

Adds mathematical, date, string, and other functions to Spark built-in functions.

Commercial use

Built-in Functions

1

DLI queue tags

Modifies the description of the queue tag key and tag value.

Commercial use

Managing Queue Tags

1

Tags for DLI database tables

Adds, modifies, and deletes tags for database tables.

Commercial use

Managing Tags for Database Tables

1

Template-related APIs

APIs for creating job templates and updating SQL templates are now available.

Commercial use

Templates

2

New datasource connection APIs

Call APIs to create, list, obtain, update, and delete datasource authentication, and create and delete routes.

Commercial use

Datasource Authentication APIs

Creating a Route

Deleting a Route

1

DLI now available in Europe

Starting today, you can use DLI in Europe.

Data Lake Insight (DLI) is a one-stop, serverless, converged big data platform that supports stream processing, batch processing, and interactive analysis. DLI frees you from managing any server.

Commercial use

Data Lake Insight (DLI)