Overview
DataArts Migration (offline jobs) uses an architecture with decoupled read and write. It supports read and write of any data sources, enabling flexible synchronization between data sources.
- Decoupled read and write of any data sources
DataArts Migration (offline jobs) can read and write data sources separately. Any data source that can be read can function as the source of data synchronization, and any data source that can be written can function as the destination of data synchronization.
You can connect any data source that can be read to any data source that can be written to form a cross-source synchronization link, for example, from a relational database to a lakehouse storage or from a file system to a real-time data warehouse.
- Differences from DataArts Migration (CDM jobs)
Compared with DataArts Migration (CDM jobs), DataArts Migration (offline jobs) uses an optimized architecture and supports cross-source synchronization with real read and write decoupling and through any link.
The traditional CDM architecture does not support read and write decoupling or synchronization between any data sources. Instead, it only supports synchronization between some predefined and fixed data sources. DataArts Migration (offline jobs) supports decoupled read and write capabilities and combination of data sources, meeting more extensive data transfer and integration requirements and adapting to various scenarios.
- Synchronization of a single table, database and table shards, and entire databases Offline data migration jobs support the following synchronization modes: from a table to another table, from a database to a table, and from a database/table shard to a table. The supported data sources vary depending on the synchronization mode.
- Single table synchronization: DataArts Migration can synchronize tables and files in data ingestion into a data lake or data migration to the cloud.
- Database and table shard synchronization: DataArts Migration can synchronize data from multiple tables in multiple databases in data ingestion into a data lake or data migration to the cloud.
- Entire database migration: DataArts Migrations can synchronize data from an on-premises database in data ingestion into a data lake or data migration to the cloud.
The supported data sources vary depending on the CDM cluster version.
Supported Data Sources
Example: If data can be read from a table on an FTP server and written to a table in Hudi, you can configure a task for synchronizing file data on the FTP server to a Hudi table.
The feasibility of cross-source synchronization depends only on whether data can be read from the source and written to the destination. The source and destination can be different types of data, and there is no limit on the source and destination pair.
Table 1 lists the data sources that support single table synchronization.
| Data Source | Single Table Read | Single Table Write | Database/Table Shard Read | Database/Table Shard Write | Entire Database Read | Entire Database Write |
|---|---|---|---|---|---|---|
| MySQL | Supported | Supported | Supported | Supported | Supported | Unsupported |
| PostgreSQL | Supported | Supported | Supported in some regions | Supported | Supported | Unsupported |
| SQL Server | Supported | Supported | Supported in some regions | Supported | Unsupported | Unsupported |
| Oracle | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| SAP HANA | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Open Gauss | Supported | Supported | Supported in some regions | Supported | Supported | Unsupported |
| GBase | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| DWS | Supported | Supported | Unsupported | Supported | Supported | Unsupported |
| DLI | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Apache Hive | Supported | Supported | Unsupported | Supported | Unsupported | Supported in some regions |
| MRS Hive | Supported | Supported | Unsupported | Supported | Unsupported | Supported in some regions |
| MRS Hudi | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Doris | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| MRS ClickHouse | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Open ClickHouse | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Fabric SQL | Unsupported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Lake Formation | Unsupported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| MRS HBase | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| MongoDB | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Redis | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| Elasticsearch | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| DMS Kafka | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| LTS | Supported in some regions | Unsupported | Unsupported | Unsupported | Unsupported | Unsupported |
| Redis | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| RestApi | Supported | Unsupported | Unsupported | Unsupported | Unsupported | Unsupported |
| OBS | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
| FTP | Supported | Supported in some regions | Unsupported | Supported in some regions | Unsupported | Unsupported |
| SFTP | Supported | Supported in some regions | Unsupported | Supported in some regions | Unsupported | Unsupported |
| Apache HDFS | Supported | Supported | Unsupported | Supported | Unsupported | Unsupported |
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
