Configuring a Real-Time Migration Job

Prerequisites

• You have authorized the use of real-time data migration.

• You have prepared data sources and the connection account has required permissions. For details, see the requirements for database account permissions in Check Before Use.
• A data connection has been created, and DataArts Migration has been selected for the connection. For details, see Creating a DataArts Studio Data Connection.

Procedure

1. Create a real-time processing migration job by referring to Creating a Real-Time Migration Job.
2. Set the data connection type.
   Select the data type of the source and that of the destination. For details about the supported source and destination data types, see Creating a Real-Time Migration Job.

   Figure 1 Selecting the data connection type
   

3. Set the migration job type.
   1. Migration Type: The default value is Real-time and cannot be changed.
   2. Migration Scenario: Select Single table, Entire DB, or Database/Table shard.
      Table 1 lists the scenarios.

      Table 1 Synchronization scenario parameters

      Scenario	Description
      Single table	A table in an instance can be synchronized to another instance.
      Entire DB	Multiple tables in multiple databases in an instance can be synchronized to another instance in real time. A task can synchronize a maximum of 200 tables.
      Database/Table shard	Multiple table shards of multiple databases in multiple instances can be synchronized to a database table in an instance.

      Figure 2 Setting the migration job type
      

4. Configure network resources.
   Select a source data connection, a destination data connection, and a migration resource group for which network connections have been configured.

   Figure 3 Selecting data connections and a migration resource group
   

   

   If no data connection is available, click Create to go to the Manage Data Connections page of the Management Center console and click Create Data Connection to create a connection. For details, see Configuring DataArts Studio Data Connection Parameters.

5. Configure source and destination parameters.

   The parameters vary depending on the source type. For details, see Tutorials.

6. Update the mapping between the source table and destination table, check whether the mapping is correct, and modify table attributes and add additional fields as required.
7. (Optional) Configure DDL message processing rules.

   Real-time migration jobs can synchronize data manipulation language (DML) operations, such as adding, deleting, and modifying data, as well as some table structure changes using the data definition language (DDL). You can set the processing policy for a DDL operation to Normal processing, Ignore, or Error.

   • Normal processing: When a DDL operation on the source database or table is detected, the operation is automatically synchronized to the destination.
   • Ignore: When a DDL operation on the source database or table is detected, the operation is ignored and not synchronized to the destination.
   • Error: When a DDL operation on the source database or table is detected, the migration job throws an exception.
     Figure 4 DDL configuration
     

8. Configure task parameters.

   Table 2 Task parameters

   Parameter	Description	Default Value
   Execution Memory	Memory allocated for job execution, which automatically changes with the number of CPU cores.	8 GB
   CPU Cores	Value range: 2 to 32 For each CPU core added, 4 GB execution memory and one concurrency are automatically added.	2
   Maximum Concurrent Requests	Maximum number of jobs that can be concurrently executed. This parameter does not need to be configured and automatically changes with the number of CPU cores.	1
   Add Custom Attribute	You can add custom attributes to modify some job parameters and enable some advanced functions. For details, see Job Performance Optimization.	N/A

9. Submit and run the job.

   After configuring the job, click Submit in the upper left corner to submit the job.

   Figure 5 Submitting the job
   

   After submitting the job, click Start on the job development page. In the displayed dialog box, set required parameters and click OK.

   Figure 6 Starting the job
   

   Table 3 Parameters for starting the job

   Parameter	Description
   Synchronization Mode	Common data synchronization modes include: Incremental synchronization: Incremental data synchronization starts from a specified time point. Full and incremental synchronization: All data is synchronized first, and then incremental data is synchronized in real time.
   	Kafka data synchronization modes include: Earliest: Data consumption starts from the earliest offset of the Kafka topic. Latest: Data consumption starts from the latest offset of the Kafka topic. Start/End time: Data consumption starts from the offset of the Kafka topic obtained based on the time.
   Time	Time when incremental synchronization starts. This parameter is mandatory when Synchronization Mode is set to Incremental synchronization or Start/End time. NOTE: If you set a time that is earlier than the earliest time of the incremental data log, data consumption starts from the latest log time by default. If you set a time earlier than the earliest offset of Kafka messages, data consumption starts from the earliest offset by default.

10. Monitor the job.

    On the job development page, click Monitor to go to the Job Monitoring page. You can view the status and log of the job, and configure alarm rules for the job. For details, see Real-Time Migration Job O&M.

    Figure 7 Monitoring the job