Creating a Real-Time Synchronization Task
Real-time synchronization refers to the real-time flow of key service data from source to destination while consistency of data can be ensured.
This section describes how to synchronize full+incremental data from an on-premises MySQL database to a Huawei Cloud RDS for MySQL instance through a public network. For more information about real-time synchronization, see Synchronization Overview.
Operation Process
|
Process |
Description |
|---|---|
|
Sign up for a HUAWEI ID, complete real-name authentication, enable Huawei Cloud services, and prepare the database to be synchronized, the database connection account, and a network. |
|
|
Select the source and destination databases, and create and start a synchronization task. |
|
|
Check the synchronization progress. |
|
|
Create a comparison task to check whether the data in the source database is consistent with that in the destination database. |
|
|
After confirming that the data is consistent, cut over workloads and stop the DRS task. |
Preparations
Before creating a real-time synchronization task, prepare the Huawei account, database to be synchronized, database connection account, and network environment by referring to the following steps.
Prepare a Huawei account, create a user, and grant permissions to the user to use DRS.
- Access Huawei Cloud website.
- Click Sign Up and follow the instructions to create your account (your HUAWEI ID).
- Select the service agreement and click Enable.
The system displays a message indicating that Huawei Cloud services have been enabled.
- Perform real-name authentication.
- Individual account: Individual Real-Name Authentication
- Enterprise account: Enterprise Real-Name Authentication
- For fine-grained permissions management, create an Identity and Access Management (IAM) user and user group on the IAM console and grant the user specific operation permissions. For details, see Creating a User and Granting Permissions
Before creating a real-time synchronization task, you need to prepare the source and destination databases.
- The source database in this example is an on-premises MySQL database. Prepare the following source database details:
Item
Example Value
Description
DB engine version
MySQL 5.7
-
IP address
10.154.217.42
Enter an IP address.
Port
3306
-
- The destination database in this example is a Huawei Cloud RDS for MySQL instance. You need to prepare the database details by referring to Getting Started with RDS for MySQL.
Item
Example Value
Description
Region
AP-Singapore
To reduce network latency, select the region nearest to you.
Instance name
rds-mysql
Specify a name that will be easy to identify.
DB engine version
MySQL 5.7
-
Instance type
Single
A single instance is used in this example.
To improve service reliability, select a primary/standby instance.
Storage type
Cloud SSD
-
AZ
AZ1
AZ1 is selected in this example.
To improve service reliability, select the primary/standby instance type and deploy the primary and standby instances in different AZs.
Instance class
General-purpose 2 vCPUs | 8 GB
-
To ensure a smooth data synchronization, you are advised to create an independent database account for connecting to the DRS task. In addition, you need to ensure that the account is granted required permissions.
- Source database user
- Log in to the source database.
- Run the following statement to create database user usersrc:
CREATE USER 'usersrc'@'%' IDENTIFIED BY 'password';
- Run the following statements to grant permissions to usersrc:
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'usersrc';
GRANT SELECT, SHOW VIEW, EVENT, LOCK TABLES, ON [Database to be synchronized].* TO 'usersrc';
- Destination database user
- Connect to an RDS for MySQL instance through DAS.
- Run the following statement to create database user usertar. For details, see Creating a Database Account Through DAS.
CREATE USER 'usertar'@'%' IDENTIFIED BY 'password';
- Run the following statement to grant permissions to usertar:
GRANT SELECT, CREATE, DROP, DELETE, INSERT, UPDATE, ALTER, REFERENCES ON *.* TO 'usertar';
Before creating a synchronization task over a public network, you need to apply for an EIP and configure the firewall of your local data center so that the EIP can access the on-premises database.
- Create an EIP for the DRS synchronization task by referring to Assigning an EIP.
- Configure the firewall of the local data center.
Add the EIP created in 1 to the whitelist of the source database to ensure that the source database can access EIP.
- Configure the IP address whitelist for the on-premises database.
Add the DRS instance EIP to the whitelist of the on-premises database to allow the access from the DRS instance.
Step 1: Create a Synchronization Task
This step describes how to set parameters based on the example values in Preparations. For more information about real-time synchronization, see Synchronization Overview.
- Go to the Create Synchronization Task page.
- Configure basic information as follows:
Figure 1 Basic information
Parameter
Example Value
Description
Billing Mode
Pay-per-use
DRS billing mode.
- Yearly/Monthly: A prepaid billing mode in which you pay for resources before using it. Bills are settled based on the subscription period.
- Pay-per-use billing: A postpaid billing mode. In this mode, you pay for what you used.
Region
AP-Singapore
The region where the current task is deployed.
Project
AP-Singapore
The project corresponds to the current region.
Task Name
DRS-Synchronization
The name of the synchronization task.
Description
Leave this parameter blank for now.
Task description.
- Configure information about the synchronization instance.
Figure 2 Synchronization instance details
Parameter
Example Value
Description
Data Flow
To the Cloud
The direction of the synchronization task. The options are To the cloud, Out of the cloud, and Self-built to self-built.
- To the cloud: The destination database is a Huawei Cloud DB instance and data needs to be transferred to the cloud.
- Out of the cloud: The source database is a Huawei Cloud DB instance and data needs to be transferred out of the cloud.
- Self-built to self-built: Neither the source database nor the destination databases is a Huawei Cloud DB instance.
Source DB Engine
MySQL
The DB engine type of the source database.
Destination DB Engine
MySQL
The DB engine type of the destination database.
Network Type
Public network
The network type of the synchronization task.
- VPC: suitable for synchronizations between cloud databases of the same account in the same region and VPC.
- Public network: suitable for synchronizations from on-premises databases or external cloud databases to destination databases.
- VPN or Direct Connect: suitable for synchronizations from on-premises databases to cloud databases or between cloud databases across regions using a VPN, Direct Connect, Cloud Connect, VPCEP, or a VPC peering connection.
DRS Task Type
Single-AZ
Type of the DRS task. The value can be Single-AZ or Dual-AZ.
Destination DB Instance
rds-mysql
Select a destination DB instance.
The destination DB instance is rds-mysql in Databases.
Synchronization Instance Subnet
Default subnet
The subnet where the synchronization task is.
Synchronization Mode
Full+Incremental
The available options are Full, Full+Incremental, and Incremental.
Enable Binlog Cleanup
No
Whether to enable the function of quickly clearing binlogs of the destination database.
Specify EIP
-
Select an EIP for the DRS instance. In this example, it is the EIP created in Network.
- Select the task specifications and AZ.
Figure 3 Specifications and AZ
Parameter
Example Value
Description
Specifications
Large
DRS instance specifications. Different specifications have different performance upper limits. For details, see Real-Time Synchronization.
AZ
AZ1
Select the AZ where you want to create the DRS task.
- Configure an enterprise project and tags.
Figure 4 Enterprise Project and Tags
Parameter
Example Value
Description
Enterprise Project
default
If your account is associated with an enterprise project, select the project from the Enterprise Project drop-down list.
For more details, see Enterprise Management User Guide.
Tags
Leave this parameter blank for now.
Tags for the synchronization task. Adding tags helps you better identify and manage your tasks.
- Click Create Now.
Wait for 5 to 10 minutes. If the message "Synchronization instance created successfully" is displayed, the DRS task is successfully created.
- On the Configure Source and Destination Databases page, specify source and destination database details and click Test Connection for both the source and destination databases to confirm they have been connected to the DRS instance.
- In the Source Database area, enter the source database details in Connection Accounts.
- Click Test Connection.
If the message "Test successful" is displayed, the source database is connected.
- In the Destination Database area, enter the destination database details in Connection Accounts.
- Click Test Connection.
If the message "Test successful" is displayed, the destination database is connected.
- Click Next.
- On the Set Synchronization Task page, select the conflict policy and synchronization objects, and then click Next.
Figure 5 Synchronization Mode
Parameter
Example Value
Description
Flow Control
No
Whether to set a maximum synchronization speed of the DRS task. This function is disabled by default. Flow Control is only used during the full synchronization phase.
Incremental Conflict Policy
Overwrite
The policy for handling data conflicts during incremental synchronization. By default, conflicts in the full synchronization phase are ignored.
Filter DROP DATABASE
No
To reduce the risks involved in a synchronization, DRS allows you to filter out DROP operations.
Synchronize
Common index and Incremental DDLs
Select the type of the objects to be synchronized.
Create Indexes Along With Table Structure
No
Whether to create indexes along with the table structure in the full synchronization phase.
Synchronize DML
Insert, Update, and Delete
Select the DML operations to be synchronized. By default, all DML operations are selected.
Online DDL
No
If table-level synchronization is selected, you can choose whether to synchronize Online DDL. By default, Online DDL is not synchronized.
Data Synchronization Topology
One-to-one
Data synchronization supports multiple synchronization topologies. You can plan your synchronization instances based on service requirements. For details, see Data Synchronization Topologies.
Synchronize DDLs
Default
Select DDL type for incremental synchronization.
Synchronization Object
Tables
Select objects to be synchronized. You can select All, Tables, or Databases as required.
- On the Process Data page, set the filtering rules for data processing.
- If data processing is required, select Data filtering, Additional Columns, or Processing Columns. For details about how to configure related rules, see Processing Data.
- In this example, data processing is not required. Click Next.
- On the Check Task page, check the synchronization task.
- If there are any items that require confirmation, view and confirm the details first before proceeding to the next step.
- If any of the checks fail, review the cause and rectify the fault. After the fault is rectified, click Check Again.
If the check success rate reaches 100%, the pre-check is successful.
- Click Next
- On the Confirm Task page, specify Start Time.
Figure 6 Task startup settings
Parameter
Example Value
Description
Start Time
Start upon task creation
Set Start Time to Start upon task creation or Start at a specified time based on site requirements. The Start at a specified time option is recommended.
Send Notifications
Disable
This parameter is optional. After enabled, select an SMN topic. If the status, latency metric, or data of the synchronization task is abnormal, DRS will send you a notification.
Stop Abnormal Tasks After
14
Any task in the Abnormal state that has run for longer than the period you set here (in days) will automatically stop.
- Confirm that the configured information is correct and click Submit to submit the task.
- After the synchronization task is submitted, go to the Data Synchronization Management page to check the task status.
- If the task status is Starting, the task has been started.
- By default, DRS retains a task in the Configuration state for three days. After three days, DRS automatically deletes background resources, but the task status remains unchanged. When you reconfigure the task, DRS applies for resources for the task again.
Step 2: Query Synchronization Progress
After the synchronization task is started, you can check the synchronization progress. DRS shows the synchronization progress using a progress bar, so you can track the synchronization progress in real time.
- On the Data Synchronization Management page, click the target synchronization task name in the Task Name/ID column.
The Basic Information page is displayed.
- Choose Synchronization Progress to check the progress and details.
- Progress: When the synchronization progress reaches 100%, full synchronization is complete. The synchronization progress is displayed based on the number of synchronized objects.
Figure 7 Synchronization progress
- Migration Details: If the values of Total Items and Synchronized Items are the same, the object synchronization is complete.
Figure 8 Synchronization progress details
- Incremental synchronization delay: After the full synchronization is complete, you can check the delay of the incremental synchronization on the Synchronization Progress page.
Delay refers to the delay from when the transaction was submitted to the source database to when it is synchronized to the destination database and executed.
If the delay is 0, the source database is consistent with the destination database, and no new transactions need to be synchronized.
- Progress: When the synchronization progress reaches 100%, full synchronization is complete. The synchronization progress is displayed based on the number of synchronized objects.
Step 3: Compare Synchronization Items
For a full+incremental synchronization task, when the full synchronization is complete and the incremental synchronization delay is 0, you can create a comparison task to check whether the data in the source database is consistent with that in the destination database.
- Object-level comparison: It compares databases, indexes, tables, views, stored procedures and functions, and sorting rules of tables.
- Data-level comparison: It checks the consistency of rows or values in the synchronization tables.
You can create an object-level comparison task to check the integrity of database objects.
- On the Data Synchronization Management page, click the target synchronization task name in the Task Name/ID column.
The Basic Information page is displayed.
- Choose Synchronization Comparison.
The Synchronization Comparison page is displayed.
- Click Object-Level Comparison.
The Object-Level Comparison page is displayed.
- Click Compare.
- After 5 to 10 minutes, click
to check the comparison result of each comparison item.
If Consistent is displayed in the Result column, the object synchronization is complete.Figure 9 Object-Level Comparison
After the database object comparison is complete, you can create a data-level comparison task to check the number of rows or values of the synchronized data.
- On the Data Synchronization Management page, click the target synchronization task name in the Task Name/ID column.
The Basic Information page is displayed.
- Choose Synchronization Comparison.
The Synchronization Comparison page is displayed.
- Click the Data-Level Comparison tab.
- Click Create Comparison Task.
- Select a comparison policy and click OK to submit the comparison task.
Figure 10 Creating a comparison task
Parameter
Example Value
Description
Comparison Type
Row
The available options are Row and Value.
- Row comparison: It checks whether the number of rows in a source table is the same as that in the synchronized table.
- Value comparison: It checks whether the data in a source table is consistent with that in the synchronized table.
Comparison Policy
One-to-one
DRS supports one-to-one and many-to-one comparison policies.
- One-to-one: compares the number of rows in a table in the source database with that in the table mapped to the destination database.
- Many-to-one: compares the number of rows in a table in the source database with that in the aggregate table mapped to the destination database.
Comparison Time
Start upon task creation
The available options are Start upon task creation and Start at a specified time.
Filter Data
Disable
Whether to set filtering criteria for comparison objects. After this function is enabled, objects can be compared based on the configured filtering criteria.
Object
Select All
Select objects to be compared as needed.
- Go back to the Data-Level Comparison tab.
- Click
to refresh the list. After the comparison task is complete, you can check the comparison result.
If Consistent is displayed in the Result column, the object data in the source database is consistent with that in the destination database.Figure 11 Row comparison result
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
