Help Center/ DataArts Studio/ User Guide/ DataArts Migration (Offline Jobs)/ Supported Data Sources/ Apache RocketMQ
Updated on 2026-05-20 GMT+08:00
View PDF
Share

Apache RocketMQ

Apache RocketMQ is an open-source distributed message middleware designed for low latency, high throughput, high availability, and strong consistency. It is widely used for asynchronous communications and decoupling in large-scale distributed systems.

DataArts Migration can migrate structured and semi-structured Apache RocketMQ data.

Preparation and Constraints

  • Network requirements

    The Apache RocketMQ data source can communicate with CDM. This ensures smooth data transmission. For details, see Enabling Network Connectivity.

  • Required permissions

    Write permission: Currently, DataArts Migration can only write data to Apache RocketMQ. When DataArts Migration functions as a producer and writes data to RocketMQ, DataArts Migration must have the Produce permission on the target topic. Otherwise, messages cannot be sent.

  • Enabling ports

    NameServer service port: 9876 (TCP) is used for data transmission between DataArts Migration and Apache RocketMQ.

Supported Migration Scenarios

DataArts Migration supports the following modes for synchronizing on-premises data:

  • Single table synchronization

    DataArts Migration supports table/file synchronization in data ingestion into a data lake or data migration to the cloud.

  • Database and table shard synchronization

    DataArts Migration supports synchronization of data from multiple databases and tables in data ingestion into a data lake or data migration to the cloud.

  • Entire DB migration

    DataArts Migrations supports synchronization of data from an on-premises database in data ingestion into a data lake or data migration to the cloud.

Database and table shard synchronization and entire DB migration are not supported in all regions. The following table lists the supported Apache RocketMQ migration scenarios.

Supported Migration Scenario

Single Table Read

Single Table Write

Database/Table Shard Read

Database/Table Shard Write

Entire DB Read

Entire DB Write

Supported

x

√

x

√

x

x

Core Capabilities

  • Connection configuration

    Configuration Item

    Supported

    Description

    AK/SK authentication

    x

    RocketMQ 4.4.0 and later versions support the access control list (ACL) mechanism and identity authentication and permission control based on AccessKey/SecretKey. Currently, it is not supported.

    SSL/TLS encrypted transmission

    x

    Enabling TLS for the broker ensures communication security. Currently, it is not supported.
  • Write capabilities

    Configuration Item

    Supported

    Description

    Data type parsing

    JSON/CSV

    Data in JSON and CSV formats can be written.

    Concurrent write

    √

    Concurrent write can fully utilize cluster resources to improve the data write speed.

    Dirty data processing

    x

    Abnormal data cannot be written to the dirty data bucket to prevent job failures caused by a small amount of abnormal data.

Creating a Data Source

Create a data source in Management Center. For details, see Configuring Data Connection Parameters.

Creating an Offline Data Migration Job

Create an Apache RocketMQ migration job in DataArts Factory. For details, see Creating an Offline Processing Migration Job.

Parent topic: Supported Data Sources