Synchronizing Data from openGauss to Hudi Using CDL

Scenario

Import data from OpenGauss to Hudi on the CDLService web UI of a cluster with Kerberos authentication enabled.

This section applies to MRS 3.3.0 and later versions.

Prerequisites

• The CDL and Hudi services have been installed in a cluster and are running properly.
• The write-ahead log function has been enabled for the OpenGauss database by referring to Enabling the Write-Ahead Log Function for the OpenGauss Database.
• You have created a human-machine user, for example, cdluser, added the user to user groups cdladmin (primary group), hadoop, kafka, and supergroup, and associated the user with the System_administrator role on FusionInsight Manager.

Procedure

1. Log in to FusionInsight Manager as user cdluser (change the password upon your first login), choose Cluster > Services > CDL, and click the link next to CDLService UI to go to the CDLService web UI.
2. Choose Link Management and click Add Link. On the displayed dialog box, set parameters for adding the opengauss and hudi links by referring to the following tables. Creating a CDL Database Connection describes the data link parameters.

   Table 1 OpenGauss data link parameters

   Parameter	Example
   Link Type	opengauss
   Name	opengausslink
   Host	100.85.xxx.xxx
   Port	8000
   DB Name	opengaussdb
   User	opengaussuser
   Password	opengauss user password
   Description	-

   Table 2 Hudi data link parameters

   Parameter	Example
   Link Type	hudi
   Name	hudilink
   Storage Type	hdfs
   Auth KeytabFile	/opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab
   Principal	cdluser
   Description	xxx

3. After the parameters are configured, click Test to check whether the data link is normal.

   After the test is successful, click OK.

4. (Optional) Choose ENV Management and click Add Env. In the displayed dialog box, configure the parameters based on the following table.

   Table 3 Parameters for adding an ENV

   Parameter	Example Value
   Name	test-env
   Driver Memory	1 GB
   Type	spark
   Executor Memory	1 GB
   Executor Cores	1
   Number Executors	1
   Queue	-
   Description	-

   Click OK.

5. Choose Job Management > Data synchronization task and click Add Job. In the displayed dialog box, set parameters. Click Next.
   The following table shows the job parameters.

   Parameter	Example
   Name	job_opengaustohudi
   Desc	-

6. Configure OpenGauss job parameters.
   1. On the Job Management page, drag the opengauss icon on the left to the editing area on the right and double-click the icon to go to the OpenGauss job configuration page. Set parameters by referring to the following table. Creating a CDL Data Synchronization Job describes the job parameters.

      Table 4 OpenGauss job parameters

      Parameter	Example
      Link	opengausslink
      Tasks Max	1
      Mode	insert, update, and delete
      dbName Alias	opengaussdb
      Slot Name	oct_twenty_two
      Slot Drop	No
      Connect With Hudi	Yes
      Topic Table Mapping	cdlschema.testtable/testtable_topic

      

   2. Click OK. The OpenGauss job parameters are configured.

7. Configure Hudi job parameters.
   1. On the Job Management page, drag the hudi icon in the Sink area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Set parameters by referring to the following table. Creating a CDL Data Synchronization Job describes the job parameters.

      Table 5 Sink Hudi job parameters

      Parameter	Example
      Link	hudilink
      Path	/cdl/test
      Interval	5
      Max Rate Per Partition	0
      Parallelism	10
      Configuring Hudi Table Attributes	Visual View
      Global Configuration of Hudi Table Attributes	-
      Configuring the Attributes of the Hudi Table: Source Table Name	cdlschema.testtable
      Configuring the Attributes of the Hudi Table: Table Type Opt Key	MERGE_ON_READ
      Configuring the Attributes of the Hudi Table: Hudi TableName Mapping	testtable or /cdlschema/testtable
      Configuring the Attributes of the Hudi Table: Hive TableName Mapping	cdlschema.testtable
      Configuring the Attributes of the Hudi Table: Table Primarykey Mapping	so_line_id,order_number
      Configuring the Attributes of the Hudi Table: Table Hudi Partition Type	time
      Configuring the Attributes of the Hudi Table: Custom Config	-

      

   2. (Optional) Select the created ENV. The default value is defaultEnv.

      

   3. Click OK.

8. Drag the two icons to associate the job parameters and click Save. The job configuration is complete.

   

9. In the job list on the Job Management page, locate the created job, click Start in the Operation column, and wait until the job is started.

   Check whether the data transmission takes effect, for example, insert data into the table in the OpenGauss database and view the content of the file imported to Hudi.