Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using CDL/ Creating a CDL Job/ Synchronizing Data from PgSQL to Hudi Using CDL
Updated on 2024-10-09 GMT+08:00

Synchronizing Data from PgSQL to Hudi Using CDL

Scenario

This section describes how to import data from PgSQL to Hudi by using the CDLService web UI of a cluster with Kerberos authentication enabled in MRS 3.2.0.

Prerequisites

  • The CDL and Hudi services have been installed in a cluster and are running properly.
  • The prerequisites for the PgSQL database have been met. For details, see Policy for Modifying Write-Ahead Logs in PostgreSQL Databases.
  • You have created a human-machine user, for example, cdluser, added the user to user groups cdladmin (primary group), hadoop, kafka, and supergroup, and associated the user with the System_administrator role on FusionInsight Manager.

Procedure

  1. Log in to FusionInsight Manager as user cdluser (change the password upon your first login), choose Cluster > Services > CDL, and click the link next to CDLService UI to go to the CDLService web UI.
  2. Choose Link Management and click Add Link. In the displayed dialog box, set parameters for adding the pgsql and hudi links by referring to the following tables. Creating a CDL Database Connection describes the data link parameters.

    Table 1 PgSQL data link parameters

    Parameter

    Example

    Link Type

    pgsql

    Name

    pgsqllink

    Host

    10.10.10.10

    Port

    5432

    DB Name

    testDB

    User

    user

    Password

    Password of the user user

    Description

    -

    Table 2 Hudi data link parameters

    Parameter

    Example

    Link Type

    hudi

    Name

    hudilink

    Storage Type

    hdfs

    Auth KeytabFile

    /opt/Bigdata/third_lib/CDL/user_libs/cdluser.keytab

    Principal

    cdluser

    Description

    xxx

  3. After the parameters are configured, click Test to check whether the data link is normal.

    After the test is successful, click OK.

  4. (Optional) Choose ENV Management and click Add Env. In the displayed dialog box, configure the parameters based on the following table.

    Table 3 Parameters for adding an ENV

    Parameter

    Example Value

    Name

    test-env

    Driver Memory

    1 GB

    Type

    spark

    Executor Memory

    1 GB

    Executor Cores

    1

    Number Executors

    1

    Queue

    -

    Description

    -

    Click OK.

  5. Choose Job Management > Data synchronization task and click Add Job. In the displayed dialog box, set parameters and click Next.

    Parameters are as follows.

    Parameter

    Example

    Name

    job_pgtohudi

    Desc

    -

  6. Configure PgSQL job parameters.

    1. On the Job Management page, drag the pgsql icon on the left to the editing area on the right and double-click the icon to go to the PgSQL job configuration page. Configure parameters based on the following table. For details about job parameters, see Creating a CDL Data Synchronization Job.
      Table 4 PgSQL job parameters

      Parameter

      Example

      Link

      pgsqllink

      Tasks Max

      1

      Mode

      insert, update, and delete

      dbName Alias

      pgsqldb

      Schema

      pgschema

      Slot Name

      pg_slot

      Enable FailOver Slot

      No

      Slot Drop

      No

      Connect With Hudi

      Yes

      Use Exist Publication

      No

      Publication Name

      publicationtest

    2. Click the plus sign (+) to display more parameters.

      • Start Time: indicates the start time of table synchronization.
      • WhiteList: Enter a table in the database.
      • Blacklist: Enter the database table that does not need to capture data.
      • Topic Table Mapping
        • This parameter is mandatory if Connect With Hudi is set to Yes.
        • Enter the table name in the first text box, for example, test. Enter a topic name in the second text box, for example, test_topic. The topic name must match the table name in the first text box.
    3. Click OK. The PgSQL job parameters are configured.

  7. Configure Hudi job parameters.

    1. On the Job Management page, drag the hudi icon in the Sink area on the left to the editing area on the right and double-click the icon to go to the Hudi job configuration page. Configure parameters based on the following table. For details about job parameters, see Creating a CDL Data Synchronization Job.
      Table 5 Sink Hudi job parameters

      Parameter

      Example Value

      Link

      hudilink

      Path

      /cdl/test

      Interval

      10

      Max Rate Per Partition

      0

      Parallelism

      10

      Target Hive Database

      default

      Configuring Hudi Table Attributes

      Visual View

      Global Configuration of Hudi Table Attributes

      -

      Configuring the Attributes of the Hudi Table: Table Name

      test

      Configuring the Attributes of the Hudi Table: Table Type Opt Key

      COPY_ON_WRITE

      Configuring the Attributes of the Hudi Table: Hudi TableName Mapping

      -

      Configuring the Attributes of the Hudi Table: Hive TableName Mapping

      -

      Configuring the Attributes of the Hudi Table: Table Primarykey Mapping

      id

      Configuring the Attributes of the Hudi Table: Table Hudi Partition Type

      -

      Configuring the Attributes of the Hudi Table: Custom Config

      -

    2. (Optional) Click the plus sign (+) to display the Execution Env parameter. Select a created environment for it. The default value is defaultEnv.

    3. Click OK.

  8. Drag the two icons to associate the job parameters and click Save. The job configuration is complete.

  9. In the job list on the Job Management page, locate the created job, click Start in the Operation column, and wait until the job is started.

    Check whether the data transmission takes effect, for example, insert data into the table in the PgSQL database and view the content of the file imported to Hudi.