Stream Ecosystem
Overview
Built on Flink and Spark, the stream ecosystem is fully compatible with the open-source Flink, Storm, and Spark APIs. It is enhanced in features and improved in performance to provide easy-to-use DLI with low latency and high throughput.
- Cloud service ecosystems
DLI can be interconnected with other services by using Stream SQLs. You can directly use SQL statements to read and write data from various cloud services, such as Data Ingestion Service (DIS), Object Storage Service (OBS), CloudTable Service (CloudTable), MapReduce Service (MRS), Relational Database Service (RDS), Simple Message Notification (SMN), and Distributed Cache Service (DCS).
- Open-source ecosystems
After connections to other VPCs are established through VPC peering connections, you can access all data sources and output targets (such as Kafka, HBase, and Elasticsearch) supported by Flink and Spark in your dedicated DLI queues.
- Custom stream ecosystems
You can compile code to obtain data from the desired cloud ecosystem or open-source ecosystem as the input data of Flink jobs.
Cloud Service Ecosystem Development
Data Source |
SQL |
Custom Stream Job |
|
Source stream that is used to obtain data from other services or databases |
Sink stream that is used to write processed data to other services or databases |
||
CloudTable |
- |
||
CSS |
- |
- |
|
DCS |
- |
||
DDS |
- |
- |
|
DIS |
- |
||
DMS |
- |
||
GaussDB(DWS) |
- |
||
MRS |
|||
OBS |
- |
||
RDS |
- |
- |
|
SMN |
- |
- |
Open-source Ecosystem Development
Data Source |
SQL |
Custom Stream Job |
|
Source Stream |
Sink Stream |
||
Apache Kafka |
- |
Custom Stream Ecosystem Development
Data Source |
SQL |
Custom Stream Job |
|
Source Stream |
Sink Stream |
||
Custom |
- |
Supported Data Formats
DLI Flink jobs support the following data formats:
Avro, Avro_merge, BLOB, CSV, EMAIL, JSON, ORC, Parquet, and XML.
Data Format |
Supported Source Stream |
Supported Sink Stream |
---|---|---|
Avro |
- |
|
Avro_merge |
- |
|
BLOB |
- |
|
CSV |
||
|
- |
|
JSON |
||
ORC |
- |
|
Parquet |
- |
|
XML |
- |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot