Stream Ecosystem

Overview

Built on Flink and Spark, the stream ecosystem is fully compatible with the open-source Flink, Storm, and Spark APIs. It is enhanced in features and improved in performance to provide easy-to-use DLI with low latency and high throughput.

The DLI stream ecosystem includes the cloud service ecosystem, open source ecosystem, and custom stream ecosystem.

Cloud service ecosystems
DLI can be interconnected with other services by using Stream SQLs. You can directly use SQL statements to read and write data from various cloud services, such as Data Ingestion Service (DIS), Object Storage Service (OBS), CloudTable Service (CloudTable), MapReduce Service (MRS), Relational Database Service (RDS), Simple Message Notification (SMN), and Distributed Cache Service (DCS).
Open-source ecosystems
After connections to other VPCs are established through VPC peering connections, you can access all data sources and output targets (such as Kafka, HBase, and Elasticsearch) supported by Flink and Spark in your dedicated DLI queues.
Custom stream ecosystems
You can compile code to obtain data from the desired cloud ecosystem or open-source ecosystem as the input data of Flink jobs.

Cloud Service Ecosystem Development

**Table 1** Development overview
Data Source	SQL		Custom Stream Job
Data Source	Source stream that is used to obtain data from other services or databases	Sink stream that is used to write processed data to other services or databases	Custom Stream Job
CloudTable	HBase Source Stream	HBase Sink Stream OpenTSDB Sink Stream	-
CSS	-	Elasticsearch Sink Stream	-
DCS	-	DCS Sink Stream	Connecting to Other Services Through Custom Stream Jobs
DDS	-	DDS Sink Stream	-
DIS	DIS Source Stream	DIS Sink Stream	-
DMS	DMS Source Stream	DMS Sink Stream	-
GaussDB(DWS)	-	GaussDB(DWS) Sink Stream (OBS Mode)	Connecting to Other Services Through Custom Stream Jobs
MRS	MRS Kafka Source Stream	MRS Kafka Sink Stream MRS HBase Sink Stream	Connecting to Other Services Through Custom Stream Jobs
OBS	OBS Source Stream	OBS Sink Stream	-
RDS	-	RDS Sink Stream	-
SMN	-	SMN Sink Stream	-

Open-source Ecosystem Development

**Table 2** Development overview
Data Source	SQL		Custom Stream Job
	Source Stream	Sink Stream
Apache Kafka	Open-Source Kafka Source Stream	Open-Source Kafka Sink Stream	-

Custom Stream Ecosystem Development

**Table 3** Development overview
Data Source	SQL		Custom Stream Job
	Source Stream	Sink Stream
Custom	Custom Source Stream	Custom Sink Stream	-

Supported Data Formats

DLI Flink jobs support the following data formats:

Avro, Avro_merge, BLOB, CSV, EMAIL, JSON, ORC, Parquet, and XML.

**Table 4** Supported source and sink streams of corresponding data formats
Data Format	Supported Source Stream	Supported Sink Stream
Avro	-	OBS Sink Stream
Avro_merge	-	OBS Sink Stream
BLOB	DIS Source Stream MRS Kafka Source Stream Open-Source Kafka Source Stream	-
CSV	DIS Source Stream OBS Source Stream Open-Source Kafka Source Stream	DIS Sink Stream OBS Sink Stream GaussDB(DWS) Sink Stream (OBS Mode) Open-Source Kafka Sink Stream File System Sink Stream
EMAIL	DIS Source Stream	-
JSON	DIS Source Stream OBS Source Stream MRS Kafka Source Stream Open-Source Kafka Source Stream	DIS Sink Stream OBS Sink Stream MRS Kafka Sink Stream
ORC	-	GaussDB(DWS) Sink Stream (OBS Mode)
Parquet	-	OBS Sink Stream File System Sink Stream
XML	DIS Source Stream	-