Help Center/ Data Lake Insight/ Service Overview/ What Is Data Lake Insight
Updated on 2024-09-29 GMT+08:00
View PDF
Share

What Is Data Lake Insight

DLI Introduction

Data Lake Insight (DLI) is a serverless data processing and analysis service fully compatible with Apache Spark, Trino, and Apache Flink ecosystems. It frees you from managing any servers.

DLI supports standard SQL and is compatible with Spark SQL and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. You can use standard SQL or Spark and Flink applications to query mainstream data formats without data ETL. DLI supports SQL statements and Spark applications for heterogeneous data sources, including RDS, GaussDB(DWS), CSS, OBS, custom databases on ECSs, and offline databases.

Functions

You can query and analyze heterogeneous data sources such as CloudTable, RDS, and GaussDB(DWS) on the cloud using access methods, such as visualized interface, RESTful API, JDBC, and Beeline. The data format is compatible with five mainstream data formats: CSV, JSON, Parquet, and ORC.

  • Basic functions
    • You can use standard SQL statements to query in SQL jobs. For details, see Spark SQL Syntax Reference.
    • Flink jobs support Flink SQL online analysis. Aggregation functions such as Window and Join, geographic functions, and CEP functions are supported. SQL is used to express service logic, facilitating service implementation. For details, see SQL Syntax Constraints and Definitions.
    • For spark jobs, fully-managed Spark computing can be performed. You can submit computing tasks through interactive sessions or in batch to analyze data in the fully managed Spark queues. For details, see SQL Syntax Constraints and Definitions.
  • Federated analysis of heterogeneous data sources
    • Spark datasource connection: Data sources such as CloudTable, GaussDB(DWS), RDS, and CSS can be accessed through DLI. For details, see Enhanced Datasource Connections.
    • Interconnection with multiple cloud services is supported in Flink jobs to form a rich stream ecosystem. The DLI stream ecosystem consists of cloud service ecosystems and open source ecosystems.
      • Cloud service ecosystem: DLI can interconnect with other services in Flink SQL. You can directly use SQL to read and write data from cloud services, such as DIS, OBS, CloudTable, MRS, RDS, SMN and DCS.
      • Open-source ecosystem: By establishing network connections with other VPCs through enhanced datasource connections, you can access all Flink and Spark-supported data sources and output sources, such as Kafka, Hbase, Elasticsearch, in the tenant-authorized DLI queues.

      For details, see Flink Jobs.

  • Storage-compute decoupling

    DLI is interconnected with OBS for data analysis. In this architecture where storage and compute are decoupled, resources of these two types are charged separately, helping you reduce costs and improving resource utilization.

    You can choose single-AZ or multi-AZ storage when you create an OBS bucket for storing redundant data on the DLI console. The differences between the two storage policies are as follows:

    • Multi-AZ storage means data is stored in multiple AZs, improving data reliability. If the multi-AZ storage is enabled for a bucket, data is stored in multiple AZs in the same region. If one AZ becomes unavailable, data can still be properly accessed from the other AZs. The multi-AZ storage is ideal for scenarios that demand high reliability. You are advised to use this policy.
    • Single-AZ storage means that data is stored in a single AZ, with lower costs.
  • Elastic resource pool

    Elastic resource pools support the CCE cluster architecture for heterogeneous resources so you can centrally manage and allocate them. For details, see Elastic Resource Pool.

    Figure 1 Elastic resource pool architecture

    Elastic resource pools have the following advantages:

    • Unified management
      • You can manage multiple internal clusters and schedule jobs. You can manage millions of cores for compute resources.
      • Elastic resource pools can be deployed across multiple AZs to support high availability.
    • Tenant resource isolation

      Resources of different queues are isolated to reduce the impact on each other.

    • Shared access and flexibility
      • Minute-level scaling helps you to handle request peaks.
      • Queue priorities and CU quotas can be set at different time to improve resource utilization.
    • Job-level isolation (supported in later versions)

      SQL jobs can run on independent Spark instances, reducing mutual impacts between jobs.

    • Automatic scaling (supported in later versions)

      The queue quota is updated in real time based on workload and priority.

    Using elastic resource pools has the following advantages.

    Advantage

    No Elastic Resource Pool

    Use Elastic Resource Pool

    Efficiency

    You need to set scaling tasks repeatedly to improve the resource utilization.

    Dynamic scaling can be done in seconds.

    Resource utilization

    Resources cannot be shared among different queues.

    For example, a queue has idle CUs and another queue is heavily loaded. Resources cannot be shared. You can only scale up the second queue.

    Queues added to the same elastic resource pool can share compute resources.

    When you set a data source, you must allocate different network segments to each queue, which requires a large number of VPC network segments.

    You can add multiple general-purpose queues in the same elastic resource pool to one network segment, simplifying the data source configuration.

    Resource allocation

    If resources are insufficient for scale-out tasks of multiple queues, some queues will fail to be scaled out.

    You can set the priority for each queue in the elastic resource pool based on the peak hours to ensure proper resource allocation.
  • BI tool

    Interconnection with Yonghong BI for data analysis. For details, see Preparing for Yonghong BI Interconnection.

DLI Core Engine: Spark+Flink+Trino

  • Spark is a unified analysis engine that is ideal for large-scale data processing. It focuses on query, compute, and analysis. DLI optimizes performance and reconstructs services based on open-source Spark. It is compatible with the Apache Spark ecosystem and interfaces, and improves performance by 2.5x when compared with open-source Spark. In this way, DLI enables you to perform query and analysis of EB's of data within hours.
  • Flink is a distributed compute engine that is ideal for batch processing, that is, for processing static data sets and historical data sets. You can also use it for stream processing, that is, processing real-time data streams and generating data results in real time. DLI enhances features and security based on the open-source Flink and provides the Stream SQL feature required for data processing.
  • Trino, previously known as PrestoSQL, is an open source SQL query engine that allows for interactive query and analysis. It excels in quickly and efficiently processing large-scale data queries and analyses with low latency.

Serverless Architecture

DLI is a serverless big data query and analysis service. It has the following advantages:

  • Pay-per-use: You pay only for what you use (scanned data volume/CUH packages). When no jobs are running, you will not be billed.
  • Auto scaling: DLI ensures you always have enough capacity on hand to deal with any traffic spikes.

Accessing DLI

A web-based service management platform is provided. You can access DLI using the management console or HTTPS-based APIs, or connect to the DLI server through the JDBC client.

  • Using the management console

    You can submit SQL, Spark, or Flink jobs on the DLI management console.

    Log in to the management console. Choose EI Enterprise Intelligence > Data Lake Insight from the service list.