Enabling Kafka High Reliability

Updated on 2024-10-09 GMT+08:00

View PDF

Scenario

To execute the CDL data synchronization tasks listed in Table 1, enable the Kafka high reliability function to prevent data loss when Kafka is faulty or restarted.

**Table 1** CDL tasks that use MRS Kafka to synchronize data
Data Source	Destination	Description
MySQL	Hudi	Synchronizes data from MySQL to Hudi.
MySQL	Kafka	Synchronizes data from MySQL to Kafka.
PgSQL	Hudi	Synchronizes data from PgSQL to Hudi.
PgSQL	Kafka	Synchronizes data from PgSQL to Kafka.
ThirdKafka	Hudi	Synchronizes data from ThirdKafka to Hudi.
ThirdKafka	Kafka	Synchronizes data from ThirdKafka to Kafka.
opengauss (supported by MRS 3.3.0 and later versions)	ThirdKafka (DMS/DRS) -> Hudi	Synchronizes data from GaussDB(for openGauss) to Hudi through ThirdKafka (DMS/DRS).
avro-oracle (supported by MRS 3.3.0 and later versions)	ThirdKafka (DMS/DRS) -> Hudi	Synchronizes data from avro-oracle to Hudi through ThirdKafka (DMS/DRS).

Prerequisites

The CDL component has been installed in an MRS cluster and is running properly.
CDL data synchronization tasks use the Kafka component.

Procedure

Log in to FusionInsight Manager and choose Cluster > Services > Kafka. Click Configurations then All Configurations.

Search for the parameters listed in Table 2 in the search box in the upper right corner and change their values.

**Table 2** Modifying Kafka parameters
Parameter	Recommended Value	Description
unclean.leader.election.enable	false	Whether a replica that is not in the ISR can be elected as the leader. If this parameter is set to true, data may be lost.
min.insync.replicas	2	Minimum number of replicas to which data is written when offsets.commit.required.acks is set to -1.