Help Center> Data Lake Insight> Developer Guide> Stream Ecosystem Development Guide
View PDF

Stream Ecosystem Development Guide

Overview

Built on Flink and Spark, the stream ecosystem is fully compatible with the open-source Flink, Storm, and Spark APIs. It is enhanced in features and improved in performance to provide easy-to-use DLI with low latency and high throughput.

The stream ecosystem development of DLI includes the cloud service ecosystem, open source ecosystem, and custom stream ecosystem.
  • Cloud service ecosystems

    DLI can be interconnected with other services by using Stream SQLs. You can directly use SQL statements to read and write data from various cloud services, such as Data Ingestion Service (DIS), Object Storage Service (OBS), CloudTable Service (CloudTable), MapReduce Service (MRS), Relational Database Service (RDS), Simple Message Notification (SMN), and Distributed Cache Service (DCS).

  • Open-source ecosystems

    After connections to other VPCs are established through VPC peering connections, you can access all data sources and output targets (such as Kafka, HBase, and Elasticsearch) supported by Flink and Spark in your dedicated DLI queues.

  • Custom stream ecosystems

    You can compile code to obtain data from the desired cloud ecosystem or open-source ecosystem as the input data of Flink jobs.

Cloud Service Ecosystem Development

Table 1 Development overview

Data Source

SQL

Custom Stream Job

Input Stream

Output Stream

CloudTable

HBase Source Stream

-

CSS

-

Elasticsearch Sink Stream

-

DCS

-

DCS Sink Stream

Connecting to Other Services Through Custom Stream Jobs

DDS

-

DDS Sink Stream

-

DIS

DIS Source Stream

DIS Sink Stream

-

DMS

DMS Source Stream

DMS Sink Stream

-

DWS

-

Connecting to Other Services Through Custom Stream Jobs

MRS

MRS Kafka Source Stream

Connecting to Other Services Through Custom Stream Jobs

OBS

OBS Source Stream

OBS Sink Stream

-

RDS

-

RDS Sink Stream

-

SMN

-

SMN Sink Stream

-

Open-source Ecosystem Development

Table 2 Development overview

Data Source

SQL

Custom Stream Job

Input Stream

Output Stream

Apache Kafka

Open-Source Kafka Source Stream

Open-Source Kafka Sink Stream

-

Custom Stream Ecosystem Development

Table 3 Development overview

Data Source

SQL

Custom Stream Job

Input Stream

Output Stream

Custom

Custom Source Stream

Custom Sink Stream

-

Supported Data Formats

DLI Flink jobs support the following data formats:

Avro, Avro_merge, BLOB, CarbonData, CSV, EMAIL, JSON, ORC, Parquet, and XML.

Table 4 Supported source and sink streams of corresponding data formats

Data Format

Supported Source Stream

Supported Sink Stream

Avro

-

OBS Sink Stream

Avro_merge

-

OBS Sink Stream

BLOB

-

CarbonData

-

OBS Sink Stream

CSV

EMAIL

DIS Source Stream

-

JSON

ORC

-

Parquet

-

XML

DIS Source Stream

-