From MySQL to DMS Kafka

This section describes how to configure the parameters of a real-time processing migration job for synchronizing data from one or more MySQL instances to a DMS Kafka database.

Scenario

Entire database: Synchronize multiple tables from a MySQL instance to multiple tables of a DMS Kafka instance in real time.
Database and table shard: Synchronize multiple table shards of multiple database shards from a MySQL instance to an aggregation table on a DMS Kafka instance.

Entire DB

Configure source parameters.
Select the MySQL tables to be migrated.
Figure 1 Selecting databases and tables

Both databases and tables can be customized. You can select one database and one table, or multiple databases and tables.
Configure destination parameters.
Figure 2 Configuring destination parameters
- Set the rule for mapping source tables and topics.
  - Destination Topic Name Rule: You can specify a single topic or built-in field for mapping source table names and destination topic names.
    The following built-in variables are available:
    
    #{source_db_name}: source database name
    
    #{source_table_name}: source table name
  - Kafka Partition Synchronization Policy: The following policies are supported. If the source does not have a primary key, data is delivered to partition 0 on the destination by default.
    - To different partitions based on the hash values of database names/table names
    - To partition 0
    - To different partitions based on the hash values of table primary keys
  - Database Operation to Be Synchronized: DDL and DML are supported.
  - Data Format in Kafka: Debezium JSON and Canal JSON are supported.
- Mapping Between Source and Destination Tables: You can change the names of mapped destination topics as needed. You can map one source topic to one destination topic or map multiple source topics to one destination topic.
  Figure 3 Configuring the mapping between source and destination tables

Database/Table Sharding

Configure source parameters.
Figure 4 Adding a logical table

Logical Table Name: Enter the name of the table to be written to DMS Kafka.

Source Database Filter: You can enter a regular expression to filter all the database shards to be written to the destination GaussDB(DWS) aggregation table.

Source Table Filter: You can enter a regular expression to filter all the table shards in the source database shard to be written to the destination GaussDB(DWS) aggregation table.

You can click Preview in the Operation column to preview an added logical table.

Preview the logical table. The more the source tables, the longer the waiting time.

Figure 5 Previewing the logical table
Configure destination parameters.
Figure 6 Configuring destination parameters
- Set the rule for mapping source tables and topics.
  - Destination Topic Name Rule: You can specify a single topic or built-in field for mapping source table names and destination topic names.
    The following built-in variables are available:
    
    #{source_db_name}: source database name
    
    #{source_table_name}: source table name
  - Kafka Partition Synchronization Policy: The following policies are supported. If the source does not have a primary key, data is delivered to partition 0 on the destination by default.
    - To different partitions based on the hash values of database names/table names
    - To partition 0
    - To different partitions based on the hash values of table primary keys
  - Database Operation to Be Synchronized: DDL and DML are supported.
  - Data Format in Kafka: Debezium JSON and Canal JSON are supported.
- Mapping Between Source and Destination Tables: You can change the names of mapped destination topics as needed. You can map one source topic to one destination topic or map multiple source topics to one destination topic.
  Figure 7 Configuring the mapping between source and destination tables