このページは、お客様の言語ではご利用いただけません。Huawei Cloudは、より多くの言語バージョンを追加するために懸命に取り組んでいます。ご協力ありがとうございました。

Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Situation Awareness
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Data Replication Service/ Service Overview/ Product Architecture and Function Principles

Product Architecture and Function Principles

Updated on 2024-11-30 GMT+08:00

The following figure shows the product architecture and function principles of DRS.

Figure 1 DRS product architecture

Architecture Description

  • Minimum permission design
    1. Java Database Connectivity (JDBC) is used to connect to the source and destination databases, so you do not have to deploy programs on the databases.
    2. A task runs on an independent and exclusively used VM. Data is isolated between tenants.
    3. The number of IP addresses is limited. Only the DRS instance IP address is allowed to access the source and destination databases.
  • Reliability design
    1. Automatic reconnection: If the connection between DRS and your database breaks down due to bad network or database switchover, DRS automatically retries the connection until the task is restored.
    2. Resumable upload: When the connection to the source or the destination is abnormal, DRS automatically marks the current replay point. After the fault is rectified, you can resume data transfer from the replay point to ensure data consistency.
    3. If the VM where the DRS replication instance is located fails, services are automatically switched to a new VM with the IP address unchanged to ensure that the migration task is not interrupted.

The character set standard used by DRS is Unicode 6.2.0.

Basic Principles of Real-Time Migration

Figure 2 Real-time migration principle
  • Take the full+incremental migration as an example. A complete migration process includes four phases.
    1. Phase 1: Structure migration. DRS queries the databases, tables, and primary keys to be migrated from the source and creates corresponding objects in the destination.
    2. Phase 2: Full data migration. DRS uses the parallel technology to query all data from the source and inserts the data into the destination, which is fast and convenient. Before the full migration is started, incremental data is extracted and saved in advance to ensure data integrity and consistency in the subsequent incremental migration process.
    3. Phase 3: Incremental data migration. After the full migration task is complete, the incremental migration task is started. The incremental data generated after the start of the full migration is continuously parsed, converted, and replayed to the destination database until data is in sync between the source and destination databases.
    4. Phase 4: To prevent data from being operated by triggers and events during the migration, triggers and events will be migrated after a migration task is complete.
  • Principles of the underlying module for full migration:

    Sharding module: calculates the sharding logic of each table using the optimized sharding algorithm.

    Extraction module: queries data from the source database in parallel mode based on the calculated shard information.

    Replay module: inserts the data queried by the extraction module into the destination database in parallel and multi-task mode.

  • Principles of the underlying module for incremental migration:

    Log reading module: reads the original incremental log data (for example, binlog for MySQL) from the source database, parses the data, converts the data into the standard log format, and stores it locally.

    Log replay module: processes and filters incremental logs based on the standard format converted by the log reading module, and synchronizes the incremental data to the destination database.

Basic Principles of Backup Migration

Figure 3 Backup migration principle

DRS allows you to migrate data from a Microsoft SQL Server database to the cloud using the backup file of the database. You can copy the full and incremental backup files of the source database to an OBS bucket. DRS downloads that files from the bucket and uploads them to the disk of the destination database. After the pre-check and verification are complete, DRS runs the import command to restore the data to the destination database.

Basic Principles of Real-Time Synchronization

Figure 4 Real-time synchronization principle

Real-time synchronization can ensure that data is always in sync between the source and destination databases. It mainly applies to synchronization from OLTP to OLAP or from OLTP to big data components in real time. The technical principles of full|+incremental synchronization and real-time migration are basically the same. However, there is a slight difference between them in different scenarios.

  1. DRS supports heterogeneous synchronization (between different DB engines). It means that DRS converts the structure definition statements of the source database to match that of the destination database. In addition, DRS can map and convert database field types. You can refer to Mapping Data Types of heterogeneous databases or use Database and Application Migration UGO (UGO) to synchronize the structure of heterogeneous databases.
  2. DRS allows you to configure data processing rules, so you can use these rules to extract, parse, and replay data to meet your service requirements.
  3. Objects such as accounts, triggers, and events cannot be synchronized.
  4. Real-time synchronization is often used in many-to-one scenario. DDL operations in many-to-one and one-to-many scenarios are specially processed.

Basic Principles of Data Subscription

Figure 5 Data subscription principle

Data subscription provides an SDK so that customers' service programs can obtain incremental data from the source database in real time.

DRS extracts original incremental logs from the source database, parses the logs into the standard format, and persists the logs to the local host. In addition, DRS invokes the notification interface of the client subscription SDK in real time to push incremental data to the client service program. Then, the client can consume data changes based on service requirements.

The incremental data consumed by the client program is recorded on the server in real time. The DRS server can continues to push incremental data from the last consumption position in scenarios such as service interruption and reconnection.

Basic Principles of Real-Time Disaster Recovery

DRS uses the real-time replication technology to implement disaster recovery for two databases. The underlying technical principles are the same as those of real-time migration. The difference is that real-time DR supports forward synchronization and backward synchronization. In addition, disaster recovery is performed on the instance-level, which means that databases and tables cannot be selected.

Basic Principles of Workload Replay

A workload replay task consists of SQL recording and replay. All of the SQL statements (create, delete, update, and query operations) executed in the required period on the source database will be downloaded by a recording tool from the binlog, and then cached and injected into the destination database where you can trigger a replay and review performance. By specifying the replay thread and speed, you can simulate the peak service load of the source database and analyze the stability of the destination database when workloads increase sharply.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback