Synchronizing Data from Hudi to DWS Using CDL

Updated on 2024-10-09 GMT+08:00

View PDF

Scenario

This section describes how to import data from Hudi to DWS by using the CDLService web UI of a cluster with Kerberos authentication enabled in MRS 3.2.0.

Prerequisites

The CDL and Hudi services have been installed in a cluster and are running properly.
The prerequisites for the DWS database have been met. For details, see Prerequisites for the DWS database.
You have created a human-machine user, for example, cdluser, added the user to user groups cdladmin (primary group), hadoop, kafka, and supergroup, and associated the user with the System_administrator role on FusionInsight Manager.

Procedure

Log in to FusionInsight Manager as user cdluser (change the password upon your first login), choose Cluster > Services > CDL, and click the link next to CDLService UI to go to the CDLService web UI.

Choose Link Management and click Add Link. In the displayed dialog box, set parameters for adding the dws and hudi links by referring to the following tables. Creating a CDL Database Connection describes the data link parameters.

**Table 1** DWS data link parameters
Parameter	Example
Link Type	dws
Name	dwstest
Host	10.10.10.10
Port	8000
DB Name	dwsdb
User	dbuser
Password	Password of the dbuser* user*
Description	-

**Table 2** Hudi data link parameters
Parameter	Example
Link Type	hudi
Name	hudilink
Storage Type	hdfs
Auth KeytabFile	/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab
Principal	cdluser
Description	xxx

After the parameters are configured, click Test to check whether the data link is normal.

After the test is successful, click OK.

(Optional) Choose ENV Management and click Add Env. In the displayed dialog box, configure the parameters based on the following table.

**Table 3** Parameters for adding an ENV
Parameter	Example Value
Name	test-env
Driver Memory	1 GB
Type	spark
Executor Memory	1 GB
Executor Cores	1
Number Executors	1
Queue	-
Description	-

Click OK.

Choose Job Management > Data synchronization task and click Add Job. In the displayed dialog box, set parameters and click Next.

Parameters are as follows.

Parameter	Example
Name	job_huditodws
Desc	-

Configure Hudi job parameters.

On the Job Management page, drag the hudi icon in the Source area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table. For details about job parameters, see Creating a CDL Data Synchronization Job.

**Table 4** Source Hudi job parameters
Parameter	Example
Link	hudilink
Interval	10
Table Info	{"table1":[{"source.database":"dwsdb","source.tablename":"tabletest","target.tablename":"tabletest"}]}

Click to enlarge

Click OK. The Hudi job parameters are configured.

Configure DWS job parameters.

On the Job Management page, drag the dws icon on the left to the editing area on the right and double-click the icon to go to the DWS job configuration page. Configure parameters based on the following table. For details about job parameters, see Creating a CDL Data Synchronization Job.

**Table 5** DWS job parameters
Parameter	Example
Link	dwstest
Query Timeout	180000
Batch Size	10

NOTE:

Data can be synchronized from Hudi to GaussDB(DWS) only when both Hudi and GaussDB(DWS) contain the precombine field.
GaussDB(DWS) tables must contain the precombine field and the primary key.
By default, the Hudi built-in field _hoodie_event_time is used. If this field is not used, enable.sink.precombine must be specified. An example is as follows:

Click OK.

Drag the two icons to associate the job parameters and click Save. The job configuration is complete.
In the job list on the Job Management page, locate the created job, click Start in the Operation column, and wait until the job is started.

Check whether the data transmission takes effect, for example, insert data into the Hudi table and view the content of the file imported to DWS.