Overview of Real-Time Jobs

The DataArts Migration module of DataArts Studio provides real-time data synchronization, which replicates data from one source to another without affecting data consistency. This function enables real-time flow of key service data.

Typical scenarios: real-time analysis, report systems, and data warehouse environments
Characteristics: Real-time synchronization meets requirements such as many-to-one and one-to-many synchronization, dynamic addition and deletion of synchronization tables, and synchronization between tables with different names.
Figure 1 How real-time synchronization works

Real-time processing migration jobs are available in Beijing4, Shanghai1, Singapore, and Guangzhou, and will be available in other regions soon. You can use this function only after you apply for the trustlist membership. To enable it, contact customer service or technical support.

Functions

Real-time migration jobs support real-time data synchronization between a wide range of data sources in various scenarios. You can synchronize multiple database tables in full or incremental mode at a time. The following figure shows the detailed functions.

Figure 2 Functions

**Table 1** Basic functions
Function	Description
Data synchronization between sources	Various types of data sources are supported. You can combine multiple input and output data sources to form a synchronization link. For details, see Supported Data Sources.
Data synchronization in a complex network environment	Data can be synchronized between cloud databases, local IDCs, and databases on ECSs. Before configuring a synchronization task, you can select a proper synchronization solution based on the network environment of the databases to ensure that the data migration resource group can communicate with the data source and destination. For details about how to enable the connectivity, see Enabling Network Communications.
Data synchronization in multiple scenarios	Real-time incremental data synchronization is supported for a table, an entire database, and database and table shards. Single table synchronization: A table in an instance can be synchronized to another instance. Entire database synchronization: Multiple tables in multiple databases in an instance can be synchronized to another instance in real time. A task can synchronize a maximum of 200 tables. Database and table shard synchronization: Multiple table shards of multiple databases in multiple instances can be synchronized to a database table in an instance.
Real-time synchronization task configuration	Real-time data synchronization can be implemented through simple visualized configuration. Customization of data source parameters Selection of source databases and tables on a GUI and matching of source databases and tables using regular expressions Customization of the matching rule between source and destination databases and tables. Field mapping: additional fields and field value assignment (constants, variables, and UDFs) Automatic table creation Definition of DDL message processing policies
Real-time synchronization task O&M	Recovering tasks upon exceptions, resuming tasks, dynamically adding or deleting tables, monitoring tasks, configuring alarms, and viewing and exporting task logs

Synchronization Scenarios

DataArts Migration supports synchronization scenarios of multiple topology types. You can plan synchronization based on your requirements.

Single table synchronization
A table in an instance can be synchronized to another instance.

Figure 3 Single table synchronization
Entire database synchronization
Multiple tables in multiple databases in an instance can be synchronized to another instance in real time. A task can synchronize a maximum of 200 tables.

Figure 4 Entire database synchronization
Database and table shard synchronization
Multiple table shards of multiple databases in multiple instances can be synchronized to a database table in an instance.

Figure 5 Database and table shard synchronization

Basic Features

Real-time data migration provides support for big data development and has the following features:

Timeliness: Data can be synchronized within seconds.
Reliability: Mechanisms such as recovery upon exceptions and automatic retry ensure data consistency and accuracy.
Diversity:
- Diverse data sources: Multiple data sources can be selected at the source and destination.
- Diverse scenarios: Some links support full and incremental synchronization, and some links support database and table shards.
Maintainability: Job monitoring and logs are supported, enabling O&M engineers to locate faults.
Ease-of-use: You only need to configure necessary information on the console.