Kafka Data

Updated on 2022-02-22 GMT+08:00

View PDF

MirrorMaker is a powerful tool for Kafka data synchronization. It is used when data needs to be synchronized between two Kafka clusters or when data in the original Kafka cluster needs to be migrated to a new Kafka cluster. MirrorMaker is a built-in tool in Kafka. It actually integrates the functions of Kafka Consumer and Producer. MirrorMaker can read data from one Kafka cluster and write the data to another Kafka cluster to implement data synchronization between Kafka clusters.

This section describes how to use the MirroMaker tool provided by MRS to synchronize and migrate Kafka cluster data. Before migrating Kafka data, ensure that the two clusters can communicate with each other by following the instructions provided in Establishing a Data Transmission Channel.

Procedure

Versions earlier than MRS 3.x:

Enable the Kerberos authentication for the clusters. For details, see Configuring Cross-Cluster Mutual Trust Relationships.
If you plan to use the MirrorMaker tool in a source cluster, go to the details page of a destination cluster and choose Components. If you plan to use the MirrorMaker tool in a destination cluster, go to the details page of a source cluster and choose Components.
Choose Kafka > Service Configuration, and change Basic to All in the parameter type drop-down box.
Click Broker > Customization and add the following rules on the displayed page:

sasl.kerberos.principal.to.local.rules = RULE:[1:$1@$0](.*@XXXYYYZZZ.COM)s/@.*//,RULE:[2:$1@$0](.*@ XXXYYYZZZ.COM)s/@.*//,DEFAULT

In the preceding rule, XXXYYYZZZ.COM indicates the domain name of the cluster (source cluster) where data resides. The domain name must be spelled in uppercase letters.
Click Save Configuration and select Restart the affected services or instances. Click Yes to restart the Kafka service.

NOTE:

For a security cluster with the Kerberos authentication enabled, perform 1 to 5. For a normal cluster with the Kerberos authentication disabled, skip 1 to 5 and go to 6.
In the cluster that uses the MirrorMaker tool, go to the cluster details page and choose .
Choose Kafka > Service Configuration, change Basic to All in the parameter type drop-down box, and change All Roles to MirrorMaker.

Parameter description:
- The bootstrap.servers parameter in the source and dest tags indicates the broker node list and port information of the source and destination Kafka clusters respectively.
- Set parameter security.protocol in the source and dest tags based on the actual configurations of the source and destination Kafka clusters.
- If the source Kafka cluster or destination Kafka cluster is a security cluster, you need to set kerberos.domain.name and sasl.kerberos.service.name in the source and dest tags. If the local host is used, you do not need to set kerberos.domain.name. If the local host is not used, set kerberos.domain.name and sasl.kerberos.service.name based on the site requirements. The default value of sasl.kerberos.service.name is kafka.
- Set whitelist in the mirror tag, that is, the name of the topic to be synchronized.
Click Save Configuration and select Restart the affected services or instances. Click Yes to restart the MirrorMaker instance.

After MirrorMaker is restarted, the data migration task is started.

Parent topic: Data Backup and Restoration

Previous topic: HBase Data

Next topic: Storage-Compute Decoupling Operation Guide

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot