Overview

Updated on 2022-07-04 GMT+08:00

View PDF

What Is DLI?

Data Lake Insight (DLI) is a Serverless data processing and analysis service fully compatible with Apache Spark and Flink ecosystems. It frees you from managing any server. DLI supports standard SQL and is compatible with Spark and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. DLI supports SQL statements and Spark applications for heterogeneous data sources, including CloudTable, RDS, DWS, CSS, OBS, custom databases on ECSs, and offline databases.

Advantages

Full SQL compatibility
You do not require a background in big data to do big data analysis. You only need to know SQL, and you are good to go. The SQL syntax is fully compatible with the standard ANSI SQL 2003.
Storage-compute decoupling
DLI decouples storage from computing, and separates application and billing, so that you can use lower costs while improving resource utilization.
Enterprise multi-tenancy
You can manage compute- or resource-related permissions by project or user, and implement fine-grained control to isolate data for each task.
O&M-free and high availability
DLI frees you from the hassle of complicated O&M and upgrade operations while you enjoy high data availability with dual-AZ deployment.

DLI Core Engine: Spark+Flink

Spark is a unified analysis engine that is ideal for large-scale data processing. It focuses on query, compute, and analysis. DLI optimizes performance and reconstructs services based on open-source Spark. It is compatible with the Apache Spark ecosystem and interfaces, and improves performance by 2.5x when compared with open-source Spark. In this way, DLI enables you to perform query and analysis of EB's of data within hours.
Flink is a distributed compute engine that is ideal for batch processing, that is, for processing static data sets and historical data sets. You can also use it for stream processing, that is, processing real-time data streams and generating data results in real time. DLI enhances features and security based on the open-source Flink and provides the Stream SQL feature required for data processing.

Serverless Architecture

DLI is a serverless big data query and analysis service. It has the following advantages:

Pay-per-use: You pay only for what you use (scanned data volume/CUH packages). When no jobs are running, you will not be billed.
Auto scaling: DLI ensures you always have enough capacity on hand to deal with any traffic spikes.

Next topic: Quick Start