What's New

1

Setting job saving policies

Whether to save job results to a DLI job bucket.

This parameter is only available for Spark SQL queues.

Once enabled, this feature cannot be disabled, and job results of a queue are consistently saved to the DLI job bucket you configure.

Commercial use

Setting Queue Properties

1

Connecting popular BI tools to DLI queues

Added support for directly connecting Superset to DLI through the driver provided by DLI.

Commercial use

Overview of Connecting a BI Tool to DLI

2

Notes and constraints on elastic resource pool scaling

Notes and constraints on scaling elastic resource pools are added.

Commercial use

Elastic Resource Pool Scaling

1

Connecting popular BI tools to DLI queues

By analyzing and processing data, DLI provides BI tools with standard, effective, and high-quality data for statistical analysis.

There are the following methods available to connect BI tools to DLI:

• DBeaver, DBT, and Yonghong BI can directly connect to DLI through the driver provided by DLI.
• Power BI, FineBI, Superset, Tableau, and Beeline can connect to DLI through Kyuubi.

Commercial use

Overview of Connecting a BI Tool to DLI

2

Developing a DLI SQL job in DataArts Studio

Huawei Cloud DataArts Studio provides a one-stop data governance platform that integrates with DLI for seamless data integration and development, enabling enterprises to manage and control their data effectively.

DLI provides the steps for developing DLI SQL jobs in DataArts Studio.

Commercial use

Developing a DLI SQL Job in DataArts Studio

1

Connecting DLI to LakeFormation

LakeFormation is an enterprise-level, all-in-one lakehouse construction service that provides unified metadata management capabilities. It can be connected in Spark and SQL job scenarios to achieve unified metadata management.

OBT

Connecting DLI to LakeFormation

2

DLI supporting HetuEngine

DLI is compatible with the HetuEngine ecosystem.

If Type is For SQL, the queue engine can be HetuEngine.

OBT

Creating a HetuEngine Queue

1

Added APIs related to resource tags

APIs related to resource tags are added to DLI. These APIs can be used to add and query resource tags.

Commercial use

Batch Adding Resource Tags

1

Optimized the content related to DLI agencies

The description of DLI agencies, including their types and use cases, is optimized.

Commercial use

DLI Agency Overview

2

Modified the method for creating compute resources in "Getting Started" in the Data Lake Insight API Reference

The method for creating compute resources is modified in "Getting Started" in the Data Lake Insight API Reference. Specifically, the procedure for creating an elastic resource pool and adding queues within it is added.

Commercial use

Creating and Submitting a SQL Job Using an API

1

Added "DLI Job Development Process" to the Data Lake Insight User Guide

The "DLI Job Development Process" section that describes operations of different types of DLI jobs is added to the Data Lake Insight User Guide.

Commercial use

DLI Job Development Process

1

Optimized the Data Lake Insight User Guide

• The DLI job development process is added.
• Preparations for using DLI are added.
• Instructions on how to migrate external data sources to DLI are added, along with migration cases in typical scenarios.
• The section "DLI Cloud Data" is added, which describes basic concepts of tables, data sources, data catalogs, and metadata.
• The structure of the manual for DLI to read and write data from and to external data sources is optimized, and instructions on how to use DEW to manage access credentials for data sources are added.
• The structure of the usage guide for different types of jobs is optimized.
• Instructions on how to use basic datasource connections and create Flink SQL jobs are taken offline.

Commercial use

DLI Job Development Process

Overview of Data Migration Scenarios

Understanding Tables, Databases, and Data Catalogs

Using DEW to Manage Access Credentials for Data Sources

1

Optimized the Data Lake Insight What's New

It is optimized as follows:

• It is adapted to the new version of the DLI engine.
• It outlines the steps for submitting jobs to queues created within a DLI elastic resource pool.

Commercial use

Using DLI to Submit a SQL Job to Query OBS Data

Using DLI to Submit a SQL Job to Query RDS for MySQL Data

Using DLI to Submit a Flink OpenSource SQL Job to Query RDS for MySQL Data

1

Commercial use

DLI Operations That Can Be Recorded by CTS

1

Added support for Spark 3.3.1

DLI follows the release consistency of the open source Spark compute engine and now supports version 3.3.1.

Commercial use

Spark Syntax Reference

2

Added support for Flink 1.15

DLI follows the release consistency of the open source Flink compute engine and now supports version 1.15.

Commercial use

Flink OpenSource SQL 1.15 Syntax Reference

3

Support for visual configuration of SQL inspection rules

DLI supports visual configuration of SQL inspection rules, providing proactive defense against typical large and low-quality SQL queries, including hint, interception, and blocking.

Commercial use

Configuring SQL Inspection Rules

4

Agency permission minimization

DLI upgrades the system agency from dli_admin_agency to dli_management_agency. The new agency includes the necessary permissions for obtaining IAM user information, datasource operations, and message notifications, effectively avoiding uncontrolled authorization issues with DLI-related services.

Commercial use

Updating Agency Permissions

1

Added notes and constraints on tag policy usage

Notes and constraints on the use of DLI tag policies are added, which means that tags are added to resources according to the tag policy rules set by the organization.

Commercial use

Tag Management

1

Added support documentation for Spark open source commands

Support documentation is added for DLI Spark open source commands.

Commercial use

Spark Open Source Commands

1

Added Spark 3.1.1 dependencies

Information on DLI Spark 3.1.1 dependencies is added.

Commercial use

Built-in Dependencies

2

Added data authorization API

A data authorization API that allows for the assignment of database or table data permissions to specified users or projects has been added.

Commercial use

Granting Users with the Data Usage Permission

3

Added monitoring metrics related to elastic resource pools

Monitoring metrics related to DLI elastic resource pools are added.

Commercial use

DLI Monitoring Metrics

1

Added the Data Lake Insight Billing

The document covers the DLI billed items, billing description, and billing examples.

Commercial use

Data Lake Insight Billing

2

Optimized the examples in the Data Lake Insight Spark SQL Syntax Reference

More examples for creating OBS and DLI tables are added to the document.

Commercial use

Creating an OBS Table Using the DataSource Syntax

Creating an OBS Table Using the Hive Syntax

Creating a DLI Table Using the DataSource Syntax

Creating a DLI Table Using the Hive Syntax

1

Viewing Total CPU Used and Output Bytes of SQL jobs

You can view Total CPU Used and Output Bytes of a SQL job whose Type is IMPORT or QUERY.

Commercial use

SQL Job Management

2

Description of the .sha256 file in the SDK installation package

You can download the .sha256 file corresponding to the SDK installation package as instructed.

Commercial use

Prerequisites

1

Allocating queues to projects

You can allocate queues to projects.

Commercial use

Allocating a Queue to an Enterprise Project

2

Managing permissions for global variables

You can manage permissions for global variables.

Commercial use

Permission Management for Global Variables

1

Added the section "Notes and Constraints" to the Data Lake Insight Service Overview

Notes and constraints on using DLI are added.

OBT

Notes and Constraints

1

Added the address for downloading Spark 3.1.1 images

You can download Spark 3.1.1 images through the address provided.

Commercial use

Spark 3.1.1 Image

2

Enhanced datasource connections

Describes the operations related to enhanced datasource connections.

Commercial use

Enhanced Datasource Connections

3

Datasource authentication

Describes the operations related to datasource authentication.

Commercial use

Datasource Authentication

4

Spark built-in functions

Adds mathematical, date, string, and other functions to Spark built-in functions.

Commercial use

Built-in Functions

1

DLI queue tags

Modifies the description of the queue tag key and tag value.

Commercial use

Managing Queue Tags

1

Tags for DLI database tables

Adds, modifies, and deletes tags for database tables.

Commercial use

Managing Tags for Database Tables

1

Template-related APIs

APIs for creating job templates and updating SQL templates are now available.

Commercial use

Template-related APIs

2

New datasource connection APIs

Call APIs to create, list, obtain, update, and delete datasource authentication, and create and delete routes.

Commercial use

Datasource Authentication APIs

Creating a Route

Deleting a Route

1

DLI now available in Europe

Starting today, you can use DLI in Europe.

Data Lake Insight (DLI) is a one-stop, serverless, converged big data platform that supports stream processing, batch processing, and interactive analysis. DLI frees you from managing any server.

Commercial use

Data Lake Insight (DLI)