What Is Data Lake Insight

Updated on 2025-01-09 GMT+08:00

View PDF

DLI Introduction

Data Lake Insight (DLI) is a serverless data processing and analysis service fully compatible with Apache Spark, HetuEngine, and Apache Flink ecosystems. It frees you from managing any servers.

DLI supports standard SQL and is compatible with Spark SQL and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. You can use standard SQL or Spark and Flink applications to query mainstream data formats without data ETL. DLI supports SQL statements and Spark applications for heterogeneous data sources, including RDS, GaussDB(DWS), CSS, OBS, custom databases on ECSs, and offline databases.

Functions

You can query and analyze heterogeneous data sources such as CloudTable, RDS, and GaussDB(DWS) on the cloud using access methods, such as visualized interface, RESTful API, JDBC, and Beeline. The data format is compatible with five mainstream data formats: CSV, JSON, Parquet, and ORC.

Basic functions
- You can use standard SQL statements to query in SQL jobs. For details, see Spark SQL Syntax Reference.
- Flink jobs support Flink SQL online analysis capabilities: supporting aggregation functions such as Window and Join, using SQL to express service logic, and achieving service implementation conveniently and quickly. For details, see Flink OpenSource SQL Syntax Reference.
- For spark jobs, fully-managed Spark computing can be performed. You can submit computing tasks through interactive sessions or in batch to analyze data in the fully managed Spark queues. For details, see SQL Syntax Constraints and Definitions.
Federated analysis of heterogeneous data sources
- Spark datasource connection: Data sources such as CloudTable, GaussDB(DWS), RDS, and CSS can be accessed through DLI. For details, see Enhanced Datasource Connections.
- Interconnection with multiple cloud services is supported in Flink jobs to form a rich stream ecosystem. The DLI stream ecosystem consists of cloud service ecosystems and open source ecosystems.
  - Cloud service ecosystem: DLI can interconnect with other services in Flink SQL. You can directly use SQL to read and write data from cloud services, such as DIS, OBS, CloudTable, MRS, RDS, SMN and DCS.
  - Open-source ecosystem: By establishing network connections with other VPCs through enhanced datasource connections, you can access all Flink and Spark-supported data sources and output sources, such as Kafka, Hbase, Elasticsearch, in the tenant-authorized DLI queues.
  For details, see Flink Jobs.
Storage-compute decoupling
DLI is interconnected with OBS for data analysis. In this architecture where storage and compute are decoupled, resources of these two types are charged separately, helping you reduce costs and improving resource utilization.

You can choose single-AZ or multi-AZ storage when you create an OBS bucket for storing redundant data on the DLI console. The differences between the two storage policies are as follows:
- Multi-AZ storage means data is stored in multiple AZs, improving data reliability. If the multi-AZ storage is enabled for a bucket, data is stored in multiple AZs in the same region. If one AZ becomes unavailable, data can still be properly accessed from the other AZs. The multi-AZ storage is ideal for scenarios that demand high reliability. You are advised to use this policy.
- Single-AZ storage means that data is stored in a single AZ, with lower costs.

Elastic resource pool

Elastic resource pools support the CCE cluster architecture for heterogeneous resources so you can centrally manage and allocate them. For details, see Elastic Resource Pool.

Figure 1 Elastic resource pool architecture

Elastic resource pools have the following advantages:

Unified management
- You can manage multiple internal clusters and schedule jobs. You can manage millions of cores for compute resources.
- Elastic resource pools can be deployed across multiple AZs to support high availability.
Tenant resource isolation
Resources of different queues are isolated to reduce the impact on each other.
Shared access and flexibility
- Minute-level scaling helps you to handle request peaks.
- Queue priorities and CU quotas can be set at different time to improve resource utilization.
Job-level isolation (supported in later versions)
SQL jobs can run on independent Spark instances, reducing mutual impacts between jobs.
Automatic scaling (supported in later versions)
The queue quota is updated in real time based on workload and priority.

Using elastic resource pools has the following advantages.

Advantage	No Elastic Resource Pool	Use Elastic Resource Pool
Efficiency	You need to set scaling tasks repeatedly to improve the resource utilization.	Dynamic scaling can be done in seconds.
Resource utilization	Resources cannot be shared among different queues. For example, a queue has idle CUs and another queue is heavily loaded. Resources cannot be shared. You can only scale up the second queue.	Queues added to the same elastic resource pool can share compute resources.
Resource utilization	When you set a data source, you must allocate different network segments to each queue, which requires a large number of VPC network segments.	You can add multiple general-purpose queues in the same elastic resource pool to one network segment, simplifying the data source configuration.
Resource allocation	If resources are insufficient for scale-out tasks of multiple queues, some queues will fail to be scaled out.	You can set the priority for each queue in the elastic resource pool based on the peak hours to ensure proper resource allocation.

BI tool
Interconnection with Yonghong BI for data analysis.

DLI Core Engine: Spark+Flink+Trino+HetuEngine

Spark is a unified analysis engine that is ideal for large-scale data processing. It focuses on query, compute, and analysis. DLI optimizes performance and reconstructs services based on open-source Spark. It is compatible with the Apache Spark ecosystem and interfaces, and improves performance by 2.5x when compared with open-source Spark. In this way, DLI enables you to perform query and analysis of EB's of data within hours.
Flink is a distributed compute engine that is ideal for batch processing, that is, for processing static data sets and historical data sets. You can also use it for stream processing, that is, processing real-time data streams and generating data results in real time. DLI enhances features and security based on the open-source Flink and provides the Stream SQL feature required for data processing.
HetuEngine is an open source SQL query engine that allows for interactive query and analysis. It excels in quickly and efficiently processing large-scale data queries and analyses with low latency.

Serverless Architecture

DLI is a serverless big data query and analysis service. It has the following advantages:

Pay-per-use: You pay only for what you use (scanned data volume/CUH packages). When no jobs are running, you will not be billed.
Auto scaling: DLI ensures you always have enough capacity on hand to deal with any traffic spikes.

Accessing DLI

A web-based service management platform is provided. You can access DLI using the management console or HTTPS-based APIs, or connect to the DLI server through the JDBC client.

Using the management console
You can submit SQL, Spark, or Flink jobs on the DLI management console.

Log in to the management console. Choose EI Enterprise Intelligence > Data Lake Insight from the service list.

Using APIs
If you need to integrate DLI into a third-party system for secondary development, you can call DLI APIs to use the service.

For details, see Data Lake Insight API Reference.
JDBC
You can use JDBC to connect to the server for data query. For details, see Obtaining the Server Connection Address.
Spark-submit
Jobs can be submitted using Spark-submit. For details, see Using Spark-submit to Submit a Spark Jar Job.

Previous topic: Infographics

Next topic: Advantages

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot