Synchronizing Data from Oracle to Hudi

Scenario

Import data from Oracle to Hudi on the CDLService web UI of a cluster with Kerberos authentication enabled.

Prerequisites

• The CDL and Hudi services have been installed in a cluster and are running properly.
• The write-ahead log function has been enabled for the Oracle database by referring to Enabling the Write-Ahead Log Function for the Oracle Database.
• You have created a human-machine user, for example, cdluser, added the user to user groups cdladmin (primary group), hadoop, kafka, and supergroup, and associated the user with the System_administrator role on FusionInsight Manager.

Procedure

1. Log in to FusionInsight Manager as user cdluser (change the password upon the first login) and choose Cluster > Services > CDL. On the Dashboard page, click the hyperlink next to CDLService UI to go to the native CDL page.
2. Choose Driver Management and click Upload Driver to upload the driver file of Oracle. For details, see Uploading a Driver File.
3. Choose Link Management and click Add Link. On the displayed dialog box, set parameters for adding the oracle and hudi links by referring to the following tables. Creating a Database Link describes the data link parameters.

   Table 1 Oracle data link parameters

   Parameter	Example Value
   Link Type	oracle
   Name	oraclelink
   DB driver	ojdbc8-12.2.0.1.jar
   Host	10.10.10.10
   Port	1521
   User	user
   Password	Password of the user user
   Sid	orcl
   Description	Data link description

   Table 2 Hudi data link parameters

   Parameter	Example Value
   Link Type	hudi
   Name	hudilink
   Storage Type	hdfs
   Auth KeytabFile	${BIGDATA_HOME}/FusionInsight_CDL_X.X.X/install/FusionInsight-CDL-X.X.X/cdl/keytabs/cdl.keytab
   Principal	cdl/test.com@HADOOP.COM
   Description	xxx

4. After the parameters are configured, click Test to check whether the data link is normal.

   After the test is successful, click OK.

5. (Optional) Choose ENV Management and click Add Env. In the displayed dialog box, configure the parameters based on the following table.

   Table 3 Parameters for adding an ENV

   Parameter	Example Value
   Name	test-env
   Driver Memory	1 GB
   Type	spark
   Executor Memory	1 GB
   Executor Cores	1
   Number Executors	1
   Queue	-
   Description	-

   Click OK.

6. Choose Job Management > Data synchronization task and click Add Job. In the displayed dialog box, set parameters. and click Next.
   Specifically:

   Parameter	Example Value
   Name	job_oracletokafka
   Desc	xxx

7. Configure Oracle job parameters.
   1. On the Job Management page, drag the oracle icon on the left to the editing area on the right and double-click the icon to go to the Oracle job configuration page. Set parameters by referring to the following table. Creating a CDL Data Synchronization Job describes the job parameters.

      Table 4 Oracle job parameters

      Parameter	Example Value
      Link	oraclelink
      Tasks Max	1
      Mode	insert, update, and delete
      Schema	ORACLEDBA
      dbName Alias	orcl
      Connect With Hudi	Yes

   2. Click the plus sign (+) to display more parameters.

      

      

      • WhiteList: Enter the name of the table in the database, for example, myclass.
      • Topic Table Mapping
        • This parameter is mandatory if Connect With Hudi is set to Yes.
        • Enter a table name in the first text box, for example, test. Enter a topic name in the second text box, for example, test_topic. The topic name must match the table name in the first text box. Enter the data filtering time in the third box.
   3. Click OK. The Oracle job parameters are configured.

8. Configure Hudi job parameters.
   1. On the Job Management page, drag the hudi icon in the Sink area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Set parameters by referring to the following table. Creating a CDL Data Synchronization Job describes the job parameters.

      Table 5 Sink Hudi job parameters

      Parameter	Example
      Link	hudilink
      Path	/cdl/test
      Interval	10
      Max Rate Per Partition	0
      Parallelism	10
      Target Hive Database	default
      Configuring Hudi Table Attributes	Visual View
      Global Configuration of Hudi Table Attributes	-
      Configuring the Attributes of the Hudi Table: Table Name	test
      Configuring the Attributes of the Hudi Table: Table Type Opt Key	COPY_ON_WRITE
      Configuring the Attributes of the Hudi Table: Hudi TableName Mapping	-
      Configuring the Attributes of the Hudi Table: Hive TableName Mapping	-
      Configuring the Attributes of the Hudi Table: Table Primarykey Mapping	id
      Configuring the Attributes of the Hudi Table: Table Hudi Partition Type	-
      Configuring the Attributes of the Hudi Table: Custom Config	-
      Execution Env	Execution Env

      

   2. Click OK.

9. After the job parameters are configured, drag the two icons to associate the job parameters and click Save. The job configuration is complete.

   

10. In the job list on the Job Management page, locate the created jobs, click Start in the Operation column, and wait until the jobs are started.

    Check whether the data transmission takes effect, for example, insert data into the table in the Oracle database and view the content of the file imported to Hudi.

    

    Currently, a transaction on the Oracle link supports a maximum of 10,000 data updates.