Help Center/ Data Lake Insight/ Getting Started/ Practices
Updated on 2025-03-17 GMT+08:00
View PDF

Practices

You can better use DLI for big data analytics and processing by following the scenario-specific instructions and best practices provided in this section.

Table 1 Common DLI development instructions and best practices

Scenario

Instructions

Description

Spark SQL job development

Using Spark SQL Jobs to Analyze OBS Data

Use a Spark SQL job to create OBS tables, and import, insert, and query OBS table data.

Flink OpenSource SQL job development

Reading Data from Kafka and Writing Data to RDS

Use a Flink OpenSource SQL job to read data from Kafka and write the data to RDS.

Reading Data from Kafka and Writing Data to GaussDB(DWS)

Use a Flink OpenSource SQL job to read data from Kafka and write the data to GaussDB(DWS).

Reading Data from Kafka and Writing Data to Elasticsearch

Use a Flink OpenSource SQL job to read data from Kafka and write the data to Elasticsearch.

Reading Data from MySQL CDC and Writing Data to GaussDB(DWS)

Use a Flink OpenSource SQL job to read data from MySQL CDC and write the data to GaussDB(DWS).

Reading Data from PostgreSQL CDC and Writing Data to GaussDB(DWS)

Use a Flink OpenSource SQL job to read data from PostgreSQL CDC and write the data to GaussDB(DWS).

Flink Jar job development

Flink Jar Job Examples

Create a custom Flink Jar job to interact with MRS.

Using Flink Jar to Write Data to OBS

Write Kafka data to OBS.

Using Flink Jar to Connect to Kafka with SASL_SSL Authentication Enabled

Use Flink OpenSource SQL to connect to Kafka with SASL_SSL authentication enabled.

Spark Jar job development

Using Spark Jar Jobs to Read and Query OBS Data

Write a Spark program to read and query OBS data, compile and package your code, and submit a Spark Jar job.