What's New
The tables below describe the functions released in each Data Lake Insight version and corresponding documentation updates. New features will be successively launched in each region.
April 2024
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Added support for Spark 3.3.1 |
DLI follows the release consistency of the open source Spark compute engine and now supports version 3.3.1. |
Commercial use |
|
|
2 |
Added support for Flink 1.15 |
DLI follows the release consistency of the open source Flink compute engine and now supports version 1.15. |
Commercial use |
|
|
3 |
Support for visual configuration of SQL inspection rules |
DLI supports visual configuration of SQL inspection rules, providing proactive defense against typical large and low-quality SQL queries, including hint, interception, and blocking. |
Commercial use |
|
|
4 |
Agency permission minimization |
DLI upgrades the system agency from dli_admin_agency to dli_management_agency. The new agency includes the necessary permissions for obtaining IAM user information, datasource operations, and message notifications, effectively avoiding uncontrolled authorization issues with DLI-related services. |
Commercial use |
March 2024
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Added constraints on tag policy usage |
DLI has added constraints on the usage of tag policies, which means that tags are added to resources according to the tag policy rules set by the organization. |
Commercial use |
February 2024
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Added support documentation for Spark open source commands |
DLI has added support documentation for Spark open source commands. |
Commercial use |
January 2024
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Added Spark 3.1.1 dependencies |
DLI has added information on Spark 3.1.1 dependencies. |
Commercial use |
|
|
2 |
Added data authorization API |
DLI has added a data authorization API that allows for the assignment of database or table data permissions to specified users or projects. |
Commercial use |
|
|
3 |
Added monitoring metrics related to elastic resource pools |
DLI has added monitoring metrics related to elastic resource pools. |
Commercial use |
November 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Optimized the examples in Data Lake Insight Spark SQL Syntax Reference |
More examples for creating OBS and DLI tables are added to the document. |
Commercial use |
Creating an OBS Table Using the DataSource Syntax Creating an OBS Table Using the Hive Syntax |
|
2 |
Added Data Lake Insight Billing |
The document covers the DLI billed items, billing description, and billing examples. |
Commercial use |
September 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Added support for viewing Total CPU Used and Output Bytes of SQL jobs |
You can view Total CPU Used and Output Bytes of a SQL job whose Type is IMPORT or QUERY. |
Commercial use |
|
|
2 |
Added the description of the .sha256 file in the SDK installation package |
Added the description of downloading the .sha256 file corresponding to the SDK installation package to Prerequisites. |
Commercial use |
August 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Permission Management for Global Variables |
Added Permission Management for Global Variables. |
Commercial use |
|
|
2 |
Added support for allocating queues to projects |
You can allocate queues to projects. |
Commercial use |
July 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Added section "Constraints" to Data Lake Insight Service Overview |
Constraints and limitations on using DLI are added. |
Commercial use |
|
|
2 |
Elastic Resource Pool |
Describes basic operations on elastic resource pools. |
Open beta testing |
June 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Spark 3.1.1 images |
Adds the address for downloading Spark 3.1.1 images. |
Commercial use |
|
|
2 |
Built-in Functions |
Adds mathematical, date, string, and other functions to Spark built-in functions. |
Commercial use |
|
|
3 |
Datasource Authentication |
Describes the operations related to datasource authentication. |
Commercial use |
|
|
4 |
Enhanced Datasource Connections |
Describes the operations related to enhanced datasource connections. |
Commercial use |
May 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Dynamic scaling for Flink jobs |
Enables dynamic scaling for Flink jobs. |
Commercial use |
|
|
2 |
Job priority |
Sets the priority of a job. |
Commercial use |
April 2023
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
DLI queue tags |
Modifies the description of the queue tag key and tag value. |
Commercial use |
December 2022
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
New datasource connection APIs |
Call APIs to create, list, obtain, update, and delete datasource authentication, and create and delete routes. |
Commercial use |
|
|
2 |
Template-related APIs |
APIs for creating job templates and updating SQL templates are now available. |
Commercial use |
October 2022
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
SNAT rules for connecting DLI queues to the public network |
Enable communications between a queue and the Internet by configuring SNAT rules and adding routes to the public network. |
Commercial use |
Configuring the Connection Between a DLI Queue and a Data Source in the Internet |
July 2022
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
New section |
Use Flink Jar to connect to a Kafka with SASL_SSL authentication enabled. |
Commercial use |
Using Flink Jar to Connect to a Kafka with SASL_SSL Authentication Enabled |
May 2022
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Flink OpenSource SQL 1.12 syntax |
DLI now supports jobs that use Flink OpenSource SQL 1.12 syntax. |
Commercial use |
April 2022
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Best practices for using CDM to migrate data to DLI |
You can use CDM to easily migrate data from other cloud services or service platforms to DLI. |
Commercial use |
March 2022
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Guide for using Spark Jar jobs to read and query OBS data |
You can find helpful guide on how to write a Spark program to read and query OBS data, compile and package your code, and submit a Spark Jar job. |
Commercial use |
|
|
2 |
Guide for using UDTFs in Spark SQL jobs |
DLI allows you to use Hive user-defined table-generating functions (UDTFs) to customize table-valued functions. Hive UDTFs are widely used in one-in-multiple-out queries and analysis. |
Commercial use |
September 2021
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Flink OpenSource SQL syntax for ClickHouse and user-defined tables |
You can refer to the syntax description for using user-defined source table and result table and ClickHouse result table in Flink OpenSource SQL jobs. |
Commercial use |
February 2021
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Multiple data versions |
DLI controls multiple versions of backup data for restoration. |
Commercial use |
|
|
2 |
Flink 1.11 |
DLI now supports Flink 1.11. |
-- |
October 2020
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Custom images |
DLI supports clusters deployed in containers. In a container cluster, components related to Spark jobs and Flink jobs run in containers. You can download custom images provided by DLI to change the container running environments of Spark jobs and Flink jobs. |
Commercial use |
August 2020
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Package billed by storage |
DLI provides storage packages for you to reduce the cost of storing data in DLI. |
Commercial use |
June 2020
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Dual-AZ compute queues |
Cross-AZ queues can quickly recover from disasters, improving computing reliability. |
Commercial use |
|
|
2 |
Spark job developer mode |
You can set parameters by calling APIs on the DLI management console. |
Commercial use |
May 2020
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
Data scanning package |
You can buy a scanning package to reduce the fee charged for the amount of data scanned by each job. |
Commercial use |
|
|
2 |
Global variables configuration |
DLI allows you to set global variables to protect key user information. |
Commercial use |
April 2020
|
No. |
Feature |
Description |
Phase |
Document |
|---|---|---|---|---|
|
1 |
IAM fine-grained authorization |
You can mange fine-grained permissions of DLI with IAM. |
Commercial use |
|
|
2 |
Flink 1.10 |
DLI now supports Flink 1.10. |
Commercial use |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
