Updated on 2024-03-14 GMT+08:00

Supported Data Sources (2.9.2.200)

CDM provides the following migration modes which support different data sources:

This section describes the data sources supported by CDM clusters of version 2.9.2.200. The supported data sources vary depending on the CDM cluster version.

Data Sources Supported by Table/File Migration

Table/File migration can migrate data in tables or files.

Table 1 describes the supported data sources.
Table 1 Supported data sources during table/file migration

Category

Source

Destination

Description

Data warehouse

GaussDB(DWS)

  • Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

The DWS physical machine management mode is not supported.

Data Lake Insight (DLI)

-

MRS ClickHouse

Data warehouse: MRS ClickHouse and Data Lake Insight (DLI)

Recommended MRS ClickHouse version: 21.3.4.X

Hadoop

MRS HDFS

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Supported by local storage. Only MRS Hive and MRS Hudi are supported in storage-compute decoupling scenarios.
  • Only MRS Hive is supported in Ranger scenarios.
  • Not supported if SSL is enabled for ZooKeeper
  • Recommended MRS HDFS versions:
    • 2.8.X
    • 3.1.X
  • Recommended MRS HBase versions:
    • 2.1.X
    • 1.3.X
  • MRS Hive and MRS Hudi 2.x versions are not supported. The following versions are recommended:
    • 1.2.X
    • 3.1.X

MRS HBase

MRS Hive

  • Data warehouse: GaussDB(DWS), Data Lake Insight (DLI), and MRS ClickHouse
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

MRS Hudi

Data warehouse: GaussDB(DWS)

FusionInsight HDFS

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • FusionInsight cannot serve as the destination.
  • Supported only by local storage and not in storage-compute decoupling scenarios
  • Not supported by Ranger
  • Not supported if SSL is enabled for ZooKeeper
  • Recommended FusionInsight HDFS versions:
    • 2.8.X
    • 3.1.X
  • Recommended FusionInsight HBase versions:
    • 2.1.X
    • 1.3.X
  • Recommended FusionInsight Hive versions:
    • 1.2.X
    • 3.1.X

FusionInsight HBase

FusionInsight Hive

Apache HBase

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Apache cannot serve as the destination.
  • Supported only by local storage and not in storage-compute decoupling scenarios
  • Not supported by Ranger
  • Not supported if SSL is enabled for ZooKeeper
  • Recommended Apache HBase versions:
    • 2.1.X
    • 1.3.X
  • Apache Hive 2.x versions are not supported. The following versions are recommended:
    • 1.2.X
    • 3.1.X
  • Recommended Apache HDFS versions:
    • 2.8.X
    • 3.1.X

Apache Hive

Apache HDFS

Object storage

Object Storage Service (OBS)

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Object Storage Migration Service (OMS) is recommended for migration between object storage services.
  • Binary files cannot be imported to a database or NoSQL.

File system

FTP

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • The file system cannot serve as the destination.
  • Only text files such as CSV files can be migrated from FTP or SFTP servers to search services. Binary files cannot.
  • obsutil is recommended for migrating data from HTTP servers to OBS. For details, see Introduction to obsutil.

SFTP

HTTP

Hadoop: MRS HDFS

Relational database

RDS for MySQL

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, MRS Hive, and MRS Hudi
  • Object storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • You are advised to use Data Replication Service (DRS) to migrate data between OLTP databases.
  • RDS for MySQL does not support the SSL mode.
  • Recommended Microsoft SQL Server version: 2005 or later
  • The KingBase database and GaussDB can be connected using the PostgreSQL connector. The supported source and destination are the same as those of the PostgreSQL data source.

RDS for SQL Server

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • Search: Elasticsearch and Cloud Search Service (CSS)

RDS for PostgreSQL

MySQL

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, MRS Hive, and MRS Hudi
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

PostgreSQL

Oracle

Microsoft SQL Server

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object-based storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

SAP HANA

  • Data warehouse: Data Lake Insight (DLI)
  • Hadoop: MRS Hive
SAP HANA data sources have the following restrictions:
  • SAP HANA cannot serve as the destination.
  • Only the 2.00.050.00.1592305219 version is supported.
  • Only the Generic Edition is supported.
  • BW/4 FOR HANA is not supported.
  • Only database names, table names, and column names consisting of English letters are supported. Special characters such as spaces and symbols are not allowed.
  • The following data types are supported: date, digit, Boolean, and character (except SHORTTEXT). Other data types such as binary are not supported.
  • During migration, tables cannot be automatically created at the destination.

Database sharding

  • Data warehouse: Data Lake Insight (DLI)
  • Hadoop: MRS HBase and MRS Hive
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • Object-based storage: Object Storage Service (OBS)

Database shards cannot serve as the destination.

A shard link connects to multiple backend data sources at the same time. The link can be used as the job source to migrate data from those data sources to other data sources.

NoSQL

Redis

Hadoop: MRS HDFS, MRS HBase, and MRS Hive

NoSQL except CloudTable cannot serve as the destination.

Document Database Service (DDS)

MongoDB

CloudTable HBase

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, MySQL, PostgreSQL, Microsoft SQL Server, and Oracle
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

Cassandra

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

Message system

Data Ingestion Service (DIS)

Search: Cloud Search Service (CSS)

The message system cannot serve as the destination.

Apache Kafka

DMS Kafka

MRS Kafka

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)
  • MRS Kafka cannot serve as the destination.
  • Supported only by local storage and not in storage-compute decoupling scenarios
  • Not supported by Ranger
  • Not supported if SSL is enabled for ZooKeeper

Search

Elasticsearch

  • Data warehouse: GaussDB(DWS) and Data Lake Insight (DLI)
  • Hadoop: MRS HDFS, MRS HBase, and MRS Hive
  • Object storage: Object Storage Service (OBS)
  • Relational database: RDS for MySQL, RDS for PostgreSQL, and RDS for SQL Server
  • NoSQL: CloudTable
  • Search: Elasticsearch and Cloud Search Service (CSS)

Only the non-security mode is supported.

Cloud Search Service (CSS)

You are advised to use Logstash to import data to CSS. For details, see Using Logstash to Import Data to Elasticsearch

In the preceding table, the non-cloud data sources, such as MySQL, include on-premises MySQL, MySQL built on ECSs, or MySQL on the third-party cloud.

Supported Data Sources in Entire DB Migration

Entire DB migration is used when an on-premises data center or a database created on an ECS needs to be synchronized to a database service or big data service on the cloud. It is suitable for offline database migration but not online real-time migration.

Table 2 lists the data sources supporting entire DB migration using CDM.

Table 2 Supported data sources in entire DB migration

Category

Data Source

Read

Write

Description

Data warehouse

Data Warehouse Service (DWS)

Supported

Supported

-

Hadoop

(available only for local storage, and not for storage-compute decoupling, Ranger, or ZooKeeper for which SSL is enabled)

MRS HBase

Supported

Supported

Entire DB migration only to MRS HBase

Recommended versions:
  • 2.1.X
  • 1.3.X

MRS Hive

Supported

Supported

Entire DB migration only to a relational database

2.x versions are not supported. The following versions are recommended:
  • 1.2.X
  • 3.1.X

FusionInsight HBase

Supported

Not supported

Recommended versions:
  • 2.1.X
  • 1.3.X

FusionInsight Hive

Supported

Not supported

Entire DB migration only to a relational database

2.x versions are not supported. The following versions are recommended:
  • 1.2.X
  • 3.1.X

Apache HBase

Supported

Not supported

Recommended versions:
  • 2.1.X
  • 1.3.X

Apache Hive

Supported

Not supported

Entire DB migration only to a relational database

2.x versions are not supported. The following versions are recommended:
  • 1.2.X
  • 3.1.X

Relational database

RDS for MySQL

Supported

Supported

Migration from OLTP to OLTP is not supported. In this scenario, you are advised to use the Data Replication Service (DRS).

RDS for PostgreSQL

Supported

Supported

RDS for SQL Server

Supported

Supported

MySQL

Supported

Not supported

PostgreSQL

Supported

Not supported

Microsoft SQL Server

Supported

Not supported

Oracle

Supported

Not supported

SAP HANA

Supported

Not supported

  • Only the 2.00.050.00.1592305219 version is supported.
  • Only the Generic Edition is supported.
  • BW/4 FOR HANA is not supported.
  • Only database names, table names, and column names consisting of English letters are supported. Special characters such as spaces and symbols are not allowed.
  • The following data types are supported: date, digit, Boolean, and character (except SHORTTEXT). Other data types such as binary are not supported.
  • During migration, tables cannot be automatically created at the destination.

Dameng database

Supported

Not supported

Only to DWS and Hive

NoSQL

Redis

Supported

Supported

-

Document Database Service (DDS)

Supported

Supported

Only migration between DDS and MRS is supported.

CloudTable Service (CloudTable)

Supported

Supported

-