Synchronizing Data from Hudi to GaussDB(DWS)

Scenario

Import data from Hudi to GaussDB(DWS) on the CDLService web UI of a cluster with Kerberos authentication enabled.

Prerequisites

• The CDL and Hudi services have been installed in a cluster and are running properly.
• The prerequisites have been enabled for the GaussDB(DWS) database by referring to Prerequisites for the GaussDB(DWS) database.
• You have created a human-machine user, for example, cdluser, added the user to user groups cdladmin (primary group), hadoop, kafka, and supergroup, and associated the user with the System_administrator role on FusionInsight Manager.

Procedure

1. Log in to FusionInsight Manager as user cdluser (change the password upon your first login), choose Cluster > Services > CDL, and click the link next to CDLService UI to go to the CDLService web UI.
2. Choose Link Management and click Add Link. On the displayed dialog box, set parameters for adding the dws and hudi links by referring to the following tables. Creating a Database Link describes the data link parameters.

   Table 1 GaussDB(DWS) data link parameters

   Parameter	Example Value
   Link Type	dws
   Name	dwstest
   Host	10.10.10.10
   Port	8000
   DB Name	dwsdb
   User	dbuser
   Password	Password of the dbuser user
   Description	-

   Table 2 Hudi data link parameters

   Parameter	Example Value
   Link Type	hudi
   Name	hudilink
   Storage Type	hdfs
   Auth KeytabFile	/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab
   Principal	cdluser
   Description	xxx

3. After the parameters are configured, click Test to check whether the data link is normal.

   After the test is successful, click OK.

4. (Optional) Choose ENV Management and click Add Env. In the displayed dialog box, configure the parameters based on the following table.

   Table 3 Parameters for adding an ENV

   Parameter	Example
   Name	test-env
   Driver Memory	1GB
   Type	spark
   Executor Memory	1GB
   Executor Cores	1
   Number Executors	1
   Queue	-
   Description	-

   Click OK.

5. Choose Job Management > Data synchronization task and click Add Job. In the displayed dialog box, set parameters and click Next.
   Parameters are as follows.

   Parameter	Example Value
   Name	job_huditodws
   Desc	-

6. Configure Hudi job parameters.
   1. On the Job Management page, drag the hudi icon in the Source area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Set parameters by referring to the following table. Creating a CDL Data Synchronization Job describes the job parameters.

      Table 4 Source Hudi job parameters

      Parameter	Example Value
      Link	hudilink
      Interval	10
      Configuring the Attributes of the Hudi Table	JSON View
      Table Info	{"table1": [{"source.database": "","source.tablename": "","target.tablename": "","target.schemaname": "","enable.sink.precombine":"true"}] }

      

      

      • Data can be synchronized from Hudi to GaussDB(DWS) only when both Hudi and GaussDB(DWS) contain the precombine field.
      • GaussDB(DWS) tables must contain the precombine field and the primary key.
      • By default, the Hudi built-in field _hoodie_event_time is used. If this field is not used, enable.sink.precombine must be specified. An example is as follows:

        

   2. Click OK. The Hudi job parameters are configured.

7. Configure GaussDB(DWS) job parameters.
   1. On the Job Management page, drag the dws icon on the left to the editing area on the right and double-click the icon to go to the GaussDB(DWS) job configuration page. Configure parameters based on the following table.

      Table 5 GaussDB(DWS) job parameters

      Parameter	Example Value
      Link	dwstest
      Query Timeout	180000
      Batch Size	10

   2. Click OK.

8. Drag the two icons to associate the job parameters and click Save. The job configuration is complete.

   

9. In the job list on the Job Management page, locate the created job, click Start in the Operation column, and wait until the job is started.

   Check whether the data transmission takes effect, for example, insert data into the Hudi table and view the content of the file imported to GaussDB(DWS).