Help Center/ Data Lake Insight/ What's New

What's New

Updated on 2024/11/06 GMT+08:00
Share

The tables below describe the functions released in each Data Lake Insight version and corresponding documentation updates. New features will be successively launched in each region.

August 2024

No.

Feature

Description

Phase

Document

1

Optimized the Data Lake Insight User Guide

  • The DLI job development process is added.

  • Preparations for using DLI are added.

  • Instructions on how to migrate external data sources to DLI are added, along with migration cases in typical scenarios.

  • The section "DLI Cloud Data" is added, which describes basic concepts of tables, data sources, data catalogs, and metadata.

  • The structure of the manual for DLI to read and write data from and to external data sources is optimized, and instructions on how to use DEW to manage access credentials for data sources are added.

  • The structure of the usage guide for different types of jobs is optimized.

  • Instructions on how to use basic datasource connections and create Flink SQL jobs are taken offline.

Commercial use

Overview of Data Migration Scenarios

Understanding Tables

Databases

and Data Catalogs

Using DEW to Manage Access Credentials for Data Sources

July 2024

No.

Feature

Description

Phase

Document

1

Optimized the Data Lake Insight What's New

It is optimized as follows:

  • It is adapted to the new version of the DLI engine.

  • It outlines the steps for submitting jobs to queues created within a DLI elastic resource pool.

Commercial use

Using DLI to Submit a SQL Job to Query OBS Data

Using DLI to Submit a SQL Job to Query RDS for MySQL Data

Using DLI to Submit a Flink OpenSource SQL Job to Query RDS for MySQL Data

May 2024

No.

Feature

Description

Phase

Document

1

Support for DLI jobs to access resources in a shared VPC

DLI now allows for establishing network connections between DLI and resources in a shared VPC, enabling access to those resources when submitting jobs.

Commercial use

Establishing a Network Connection Between DLI and Resources in a Shared VPC

2

Added audit events related to Flink OpenSource SQL jobs

Audit events related to Flink OpenSource SQL jobs are added.

Commercial use

DLI Operations That Can Be Recorded by CTS

April 2024

No.

Feature

Description

Phase

Document

1

Added support for Spark 3.3.1

DLI follows the release consistency of the open source Spark compute engine and now supports version 3.3.1.

Commercial use

Spark Syntax Reference

2

Added support for Flink 1.15

DLI follows the release consistency of the open source Flink compute engine and now supports version 1.15.

Commercial use

Flink OpenSource SQL 1.15 Syntax Reference

3

Support for visual configuration of SQL inspection rules

DLI supports visual configuration of SQL inspection rules, providing proactive defense against typical large and low-quality SQL queries, including hint, interception, and blocking.

Commercial use

Configuring SQL Inspection Rules

4

Agency permission minimization

DLI upgrades the system agency from dli_admin_agency to dli_management_agency. The new agency includes the necessary permissions for obtaining IAM user information, datasource operations, and message notifications, effectively avoiding uncontrolled authorization issues with DLI-related services.

Commercial use

Updating Agency Permissions

March 2024

No.

Feature

Description

Phase

Document

1

Added constraints on tag policy usage

DLI has added constraints on the usage of tag policies, which means that tags are added to resources according to the tag policy rules set by the organization.

Commercial use

Tag Management

February 2024

No.

Feature

Description

Phase

Document

1

Added support documentation for Spark open source commands

DLI has added support documentation for Spark open source commands.

Commercial use

Spark Open Source Commands

January 2024

No.

Feature

Description

Phase

Document

1

Added Spark 3.1.1 dependencies

DLI has added information on Spark 3.1.1 dependencies.

Commercial use

Built-in Dependencies

2

Added data authorization API

DLI has added a data authorization API that allows for the assignment of database or table data permissions to specified users or projects.

Commercial use

Granting Users with the Data Usage Permission

3

Added monitoring metrics related to elastic resource pools

DLI has added monitoring metrics related to elastic resource pools.

Commercial use

DLI Monitoring Metrics

November 2023

No.

Feature

Description

Phase

Document

1

Optimized the examples in Data Lake Insight Spark SQL Syntax Reference

More examples for creating OBS and DLI tables are added to the document.

Commercial use

Creating an OBS Table Using the DataSource Syntax

Creating an OBS Table Using the Hive Syntax

Creating a DLI Table Using the DataSource Syntax

Creating a DLI Table Using the Hive Syntax

2

Added Data Lake Insight Billing

The document covers the DLI billed items, billing description, and billing examples.

Commercial use

Data Lake Insight Billing

September 2023

No.

Feature

Description

Phase

Document

1

Added support for viewing Total CPU Used and Output Bytes of SQL jobs

You can view Total CPU Used and Output Bytes of a SQL job whose Type is IMPORT or QUERY.

Commercial use

SQL Job Management

2

Added the description of the .sha256 file in the SDK installation package

Added the description of downloading the .sha256 file corresponding to the SDK installation package to Prerequisites.

Commercial use

Preparing the SDK Environment

August 2023

No.

Feature

Description

Phase

Document

1

Permission Management for Global Variables

Added Permission Management for Global Variables.

Commercial use

Permission Management for Global Variables

2

Added support for allocating queues to projects

You can allocate queues to projects.

Commercial use

Allocating a Queue to an Enterprise Project

July 2023

No.

Feature

Description

Phase

Document

1

Added section "Constraints" to Data Lake Insight Service Overview

Constraints and limitations on using DLI are added.

Commercial use

Constraints

2

Elastic Resource Pool

Describes basic operations on elastic resource pools.

Open beta testing

Elastic Resource Pool

June 2023

No.

Feature

Description

Phase

Document

1

Spark 3.1.1 images

Adds the address for downloading Spark 3.1.1 images.

Commercial use

Spark 3.1.1 images

2

Built-in Functions

Adds mathematical, date, string, and other functions to Spark built-in functions.

Commercial use

Built-in Functions

3

Datasource Authentication

Describes the operations related to datasource authentication.

Commercial use

Datasource Authentication

4

Enhanced Datasource Connections

Describes the operations related to enhanced datasource connections.

Commercial use

Enhanced Datasource Connections

May 2023

No.

Feature

Description

Phase

Document

1

Dynamic scaling for Flink jobs

Enables dynamic scaling for Flink jobs.

Commercial use

Dynamic scaling for Flink jobs

2

Job priority

Sets the priority of a job.

Commercial use

Job priority

April 2023

No.

Feature

Description

Phase

Document

1

DLI queue tags

Modifies the description of the queue tag key and tag value.

Commercial use

DLI queue tags

December 2022

No.

Feature

Description

Phase

Document

1

New datasource connection APIs

Call APIs to create, list, obtain, update, and delete datasource authentication, and create and delete routes.

Commercial use

Datasource Authentication APIs

Creating a Route

Creating a Route

2

Template-related APIs

APIs for creating job templates and updating SQL templates are now available.

Commercial use

Templates

October 2022

No.

Feature

Description

Phase

Document

1

SNAT rules for connecting DLI queues to the public network

Enable communications between a queue and the Internet by configuring SNAT rules and adding routes to the public network.

Commercial use

Configuring the Connection Between a DLI Queue and a Data Source in the Internet

July 2022

No.

Feature

Description

Phase

Document

1

New section

Use Flink Jar to connect to a Kafka with SASL_SSL authentication enabled.

Commercial use

Using Flink Jar to Connect to a Kafka with SASL_SSL Authentication Enabled

May 2022

No.

Feature

Description

Phase

Document

1

Flink OpenSource SQL 1.12 syntax

DLI now supports jobs that use Flink OpenSource SQL 1.12 syntax.

Commercial use

Flink OpenSource SQL 1.12 Syntax

April 2022

No.

Feature

Description

Phase

Document

1

Best practices for using CDM to migrate data to DLI

You can use CDM to easily migrate data from other cloud services or service platforms to DLI.

Commercial use

Migrating Data from Hive to DLI

March 2022

No.

Feature

Description

Phase

Document

1

Guide for using Spark Jar jobs to read and query OBS data

You can find helpful guide on how to write a Spark program to read and query OBS data, compile and package your code, and submit a Spark Jar job.

Commercial use

Using Spark Jar Jobs to Read and Query OBS Data

2

Guide for using UDTFs in Spark SQL jobs

DLI allows you to use Hive user-defined table-generating functions (UDTFs) to customize table-valued functions. Hive UDTFs are widely used in one-in-multiple-out queries and analysis.

Commercial use

Calling UDTFs in Spark SQL Jobs

September 2021

No.

Feature

Description

Phase

Document

1

Flink OpenSource SQL syntax for ClickHouse and user-defined tables

You can refer to the syntax description for using user-defined source table and result table and ClickHouse result table in Flink OpenSource SQL jobs.

Commercial use

User-defined Source Table

User-defined Result Table

ClickHouse Result Table

February 2021

No.

Feature

Description

Phase

Document

1

Multiple data versions

DLI controls multiple versions of backup data for restoration.

Commercial use

Enabling or Disabling Multiversion Backup

2

Flink 1.11

DLI now supports Flink 1.11.

--

Apache Flink Documentation

October 2020

No.

Feature

Description

Phase

Document

1

Custom images

DLI supports clusters deployed in containers. In a container cluster, components related to Spark jobs and Flink jobs run in containers. You can download custom images provided by DLI to change the container running environments of Spark jobs and Flink jobs.

Commercial use

Overview of Custom Images

August 2020

No.

Feature

Description

Phase

Document

1

Package billed by storage

DLI provides storage packages for you to reduce the cost of storing data in DLI.

Commercial use

Billing

June 2020

No.

Feature

Description

Phase

Document

1

Dual-AZ compute queues

Cross-AZ queues can quickly recover from disasters, improving computing reliability.

Commercial use

Queue Management Overview

2

Spark job developer mode

You can set parameters by calling APIs on the DLI management console.

Commercial use

Creating a Batch Processing Job

May 2020

No.

Feature

Description

Phase

Document

1

Data scanning package

You can buy a scanning package to reduce the fee charged for the amount of data scanned by each job.

Commercial use

Billing

2

Global variables configuration

DLI allows you to set global variables to protect key user information.

Commercial use

Global Variables

April 2020

No.

Feature

Description

Phase

Document

1

IAM fine-grained authorization

You can mange fine-grained permissions of DLI with IAM.

Commercial use

Creating a Custom Policy

2

Flink 1.10

DLI now supports Flink 1.10.

Commercial use

Apache Flink Documentation

Show allHide