Synchronizing Data from Hudi to ClickHouse Using CDL

Updated on 2024-10-09 GMT+08:00

View PDF

Scenario

This section describes how to import data from Hudi to ClickHouse by using the CDLService web UI of a cluster with Kerberos authentication enabled in MRS 3.2.0.

Prerequisites

The CDL, Hudi, and ClickHouse services have been installed in a cluster and are running properly.
You have operation permissions on ClickHouse. For details, see Creating a ClickHouse Role.
You have created a human-machine user with the ClickHouse administrator permissions (for details, see Creating a ClickHouse Role), for example, cdluser, added the user to user groups cdladmin (primary group), hadoop, kafka, and supergroup, and associated the user with the System_administrator role on FusionInsight Manager.
You have manually created a local table and distributed table on ClickHouse. The local table uses the ReplicatedReplacingMergeTree engine. For details, see ClickHouse Client Practices.

Procedure

Log in to FusionInsight Manager as user cdluser (change the password upon your first login), choose Cluster > Services > CDL, and click the link next to CDLService UI to go to the CDLService web UI.

Choose Link Management and click Add Link. In the displayed dialog box, set parameters for adding the clickhouse and hudi links by referring to the following tables. Creating a CDL Database Connection describes the data link parameters.

**Table 1** ClickHouse data link parameters
Parameter	Example
Link Type	clickhouse
Name	cklink
Host	10.10.10.10:21428
User	cdluser
Password	Password of the cdluser* user*
Description	-

**Table 2** Hudi data link parameters
Parameter	Example
Link Type	hudi
Name	hudilink
Storage Type	hdfs
Auth KeytabFile	/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab
Principal	cdluser
Description	-

After the parameters are configured, click Test to check whether the data link is normal.

After the test is successful, click OK.

(Optional) Choose ENV Management and click Add Env. In the displayed dialog box, configure the parameters based on the following table.

**Table 3** Parameters for adding an ENV
Parameter	Example Value
Name	test-env
Driver Memory	1 GB
Type	spark
Executor Memory	1 GB
Executor Cores	1
Number Executors	1
Queue	-
Description	-

Click OK.

Choose Job Management > Data synchronization task and click Add Job. In the displayed dialog box, set parameters and click Next.

Parameters are as follows.

Parameter	Example
Name	job_huditock
Desc	-

Configure Hudi job parameters.

On the Job Management page, drag the hudi icon in the Source area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table. For details about job parameters, see Creating a CDL Data Synchronization Job.

**Table 4** Source Hudi job parameters
Parameter	Example
Link	hudilink
Interval	10
Table Info	{"table1":[{"source.database":"db","source.tablename":"tabletest","target.tablename":"default.tabletest"}]} NOTE: You do not need to configure the fields provided by Hudi. You only need to configure the service fields that need to be synchronized to ClickHouse.

Click to enlarge

Click OK. The Hudi job parameters are configured.

Configure ClickHouse job parameters.

On the Job Management page, drag the clickhouse icon on the left to the editing area on the right and double-click the icon to go to the ClickHouse job configuration page. Configure parameters based on the following table. For details about job parameters, see Creating a CDL Data Synchronization Job.

**Table 5** ClickHouse job parameters
Parameter	Example
Link	cklink
Query Timeout	60000
Batch Size	100

Click to enlarge

Click OK.

Drag the two icons to associate the job parameters and click Save. The job configuration is complete.
In the job list on the Job Management page, locate the created job, click Start in the Operation column, and wait until the job is started.

Check whether the data transmission takes effect, for example, insert data into the Hudi table and view the content of the file imported to ClickHouse.