What Are Possible Causes of High Latency in DRS Incremental Phase?

Symptom

In the incremental migration or synchronization, the task latency is high.

Possible Causes

• Cause 1: The full phase is just complete, and the incremental migration delay is long. During a full migration, incremental data is continuously written to the source database. DRS synchronizes the incremental data to the destination database after the full migration is complete. The latency is high.
• Cause 2: Large volumes of data is imported, updated, or deleted in the source database. It takes a long time to write a large transaction to the source database, and it takes a period of time for DRS to synchronize data to the destination database. The latency increases gradually. If the table of the large transaction does not have a primary key or index, the recovery time is prolonged.
• Cause 3: A lot of DDL operations are performed in the source database. As a result, the latency increases.
• Cause 4: The DRS task specifications are limited. Different DRS specifications correspond to different performance upper limits. When the amount of data written to the source database reaches the bottleneck, tasks will be delayed.
• Cause 5: The class of the destination database is limited, reaching the write bottleneck. For example, if the destination database is RDS for MySQL, you can view database performance metrics on the RDS console.
• Cause 6: There may be hotspot updates. Writing data to a table without a primary key causes hotspot updates. Frequent updates of a single table or row in the source database also cause hotspot updates, increasing the latency. Take RDS for MySQL as an example. You can check the RDS audit logs.
• Cause 7: The network is unstable.

Solution

• Solution 1: In this case, DRS automatically adjusts the latency to a normal value. No action is required. You can check whether the incremental latency decreases.
• Solution 2: If a large transaction is written, wait until the update is complete or avoid writing a large transaction. You can view the execution history of the source database to check whether large transactions are written. Also, you can view the DRS data replay in the destination database. Take MySQL as an example. Run the show processlist command.
• Solution 3: Do not execute DDL statements in batches in the source database. If required, execute them during off-peak hours.
• Solution 4: Create a synchronization task again and select a larger specification to improve the synchronization performance. (Currently, DRS supports specification upgrade only in MySQL-to-MySQL synchronization tasks with single-node DRS instances configured. Task specifications cannot be downgraded.)
• Solution 5: Upgrade the instance class of the destination database to improve the write performance.
• Solution 6: If there are hotspot updates, wait until the hotspot updates are complete or avoid hotspot updates.
• Solution 7: Access the source and destination databases through Direct Connect to reduce latency.