Help Center/ Data Replication Service/ Best Practices/ Real-Time Synchronization/ From RDS MySQL to Kafka/ Creating a DRS Synchronization Task
Updated on 2022-09-26 GMT+08:00
View PDF

Creating a DRS Synchronization Task

This section describes how to create a DRS instance and synchronize incremental data from RDS MySQL to Kafka.

Pre-Check

Before creating a task, check the synchronization conditions.

In this example, data is synchronized from RDS MySQL to Kafka. For details, see Precautions.

Procedure

This section describes how to perform incremental synchronization from RDS MySQL to Kafka.

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select EU-Dublin.
  3. Under the service list, choose Databases > Data Replication Service.
  4. Choose Data Synchronization Management and click Create Synchronization Task.
  5. Configure synchronization task parameters.

    1. Task name

    2. Select the source database, destination database, and network information.

      Select the RDS instance created in Preparing for Source RDS MySQL as the destination database.

    3. Set Enterprise Project to default.

  6. Click Next.

    The synchronization instance is being created. It takes about 5 to 10 minutes.

  7. Configure source and destination database information.

    1. Configure source database information.
    2. Click Test Connection.

      If a successful test message is returned, login to the destination is successful.

    3. Select the VPC and subnet where the destination database is located, and enter the Kafka IP address and port number.
    4. Click Test Connection.

      If a successful test message is returned, login to the destination is successful.

  8. Click Next.
  9. Select the synchronization information, policy, message format, and object, and the format of the message sent to the Kafka.

    The following table lists the settings.

    Table 1 Sync Settings

    Type

    Setting

    Topic Synchronization Policy

    Deliver the content to a topic named testTopic.

    Synchronize Topic To

    Partitions are differentiated by the hash values of the database and table names

    Data Format in Kafka

    You can select the JSON format. For details, see Kafka Message Format.

    Synchronization Object

    Select table3_ under db_test.

  10. Click Next.
  11. Select a data processing mode. Data synchronization from RDS MySQL to Kafka supports only column-based processing, which provides column-level query and filtering capabilities.

  12. Click Next and wait for the check results.
  13. If the check is complete and the check success rate is 100%, click Next.

  14. After confirming that the synchronization task information is correct, click Next.

    Return to the Data Synchronization Management page and check the synchronization task status.

    It takes several minutes to complete.

    If the status changes to Incremental synchronization, the synchronization task has been started.

    • RDS for MySQL supports only incremental synchronization to Kafka. After the task is started, it is in the incremental synchronization state.
    • If the created task is a full synchronization task, full synchronization is performed after the task is started. After the synchronization is complete, the task automatically stops.
    • If the created task is a full+incremental synchronization task, the task enters the full synchronization state after being started. After the full synchronization is complete, the task enters the incremental synchronization state.
    • Incremental synchronization continuously synchronizes incremental data and does not stop automatically.

Parent topic: From RDS MySQL to Kafka