RocketMQ-to-RocketMQ Data Synchronization
Constraints
- RocketMQ stream synchronization supports only normal and ordered messages. Messages of other types will not be synchronized.
- If the source is a RocketMQ instance:
- The source and target must be DMS RocketMQ instances of the same version. The supported versions are 4.8.0 and 5.x.
- The broker quantity, broker flavor, and storage space of the target RocketMQ instance must be the same as those of the source RocketMQ instance.
- The instance type (for example, single-node or cluster) of the target RocketMQ instance must be the same as that of the source RocketMQ instance.
- If the source is a RocketMQ address:
- The source RocketMQ must support the query of cluster information and topic list. Otherwise, the online synchronization will fail.
- The target must be a DMS RocketMQ instance of version 4.8.0 or 5.x. The source must be a DMS RocketMQ instance of version 4.8.0 or 5.x, or a cloud vendor's or self-built RocketMQ instance that is compatible with open-source RocketMQ 4.x or 5.x.
- The number of brokers of the target RocketMQ instance must be the same as that of the source RocketMQ instance. The broker specifications, number of queues, and storage space of the target RocketMQ instance must be greater than or equal to those of the source RocketMQ instance.
Prerequisites
- The source and target RocketMQ instances are available.
- Ensure that the VPC, subnet, source and target RocketMQ topic queues, and storage space are sufficient.
- Ensure that the source and target instances can communicate with the VPC selected when you create the event stream cluster.
Procedure
- Log in to the EG console.
- In the navigation pane, choose Event Streams > Professional Event Stream Jobs.
- Click Create Job in the upper right corner. The Basic Settings page is displayed.
- Configure the basic settings. The following uses a non-whitelisted account as an example.
Table 1 Basic information parameters Parameter
Description
Cluster
Select a created cluster. If no cluster is created, create one by referring to Professional Event Stream Clusters.
Job Name
Enter a job name.
Scenario
Sync is selected by default.
Data across sources can be synchronized in real time.
Description
Enter the description of the job.
- Click Next: Configure Source and Target. The Configure Source and Target page is displayed.
Table 2 Parameters for configuring source and target data Parameter
Description
Type
Select the configuration type. Options: RocketMQ instance and RocketMQ address. The target data type is RocketMQ instance by default.
Instance Alias
Enter an instance alias, which
identifies the source and target instance. You are advised to set only one alias for the same source or target instance.
Region
Select a region.
Project
Select a project.
Instance
Select a RocketMQ instance.
Username
Enter a username.
Secret Key
Enter a secret key.
NameServer Address
When Type is set to RocketMQ address, this parameter must be set in the source data.
Enter the NameServer addresses.
Broker Address
When Type is set to RocketMQ address, this parameter must be set in the source data.
You can click Auto Obtain to obtain the broker addresses if you have entered the NameServer addresses. Note: This feature requires cluster information query in the source. If it fails, manually enter the broker addresses.
SSL
When Type is set to RocketMQ address, this parameter must be set in the source data.
Set whether SSL is enabled.
ACL
When Type is set to RocketMQ address, this parameter must be set in the source data.
Indicates whether access control is enabled.
- Click Test Connectivity. After confirming that the instance connectivity of the source and target is normal, click Next: Advanced Settings. The Advanced Settings page is displayed.
Figure 1 Advanced settings
Table 3 Configuration parameters Parameter
Description
Topic Type
If Type is set to RocketMQ instance, select Regular expression or Exact match.
If Type is set to RocketMQ address, select Regular expression or Enter.
NOTE:- If you select Regular expression, enter a regular expression in the Topics (Regular) text box. For example, .* indicates that all topics are matched, and topic.* indicates that all topics with the topic prefix are matched.
- If you select Exact match, you need to select topics.
- If you select Enter, you need to enter topics.
Sync Consumer Offset
Select whether to enable this function.
NOTE:If this function is enabled, the consumer offset will be synchronized to the target RocketMQ.
Start Offset
Select Earliest, Latest, or Custom.
Compression Algorithm
Select none, lz4, zstd, or zlib as the compression algorithm.
- Click Next: Pre-check. On the displayed page, click Finish.
- Return to the professional event stream job list and click the name of the created event stream. Click Job Management to view the synchronization details.
Table 4 Parameter description Parameter
Description
Topic
Topic created when a RocketMQ instance is created.
Queues
Number of queues set when a topic is created.
Messages to Sync
Number of unsynchronized messages in the topic queues.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot