Migrating Kafka Services

Overview

Kafka service migration refers to the process of migrating the production and consumption message clients connected to other Kafka services to ROMA Connect and migrating some persistent message files to ROMA Connect.

During service migration, services that have high requirements on continuity must be smoothly migrated to the cloud because they cannot afford a long downtime.

Preparations

1. Ensure that the network connections between the message production and consumption clients and the MQS connection address of the ROMA Connect instance are normal. You can view the MQS connection address on the Instance Information page of the ROMA Connect console.
   • If a private IP address is used for the connection, the clients and the ROMA Connect instance must be in the same VPC. If the clients and the ROMA Connect instance are in different VPCs, you can create a VPC peering connection to enable communication between the two VPCs. For details, see VPC Peering Connection.
   • If a public network address is used for connection, the clients must have the permission to access the public network.
2. Ensure that the MQS specifications of the ROMA Connect instance cannot be lower than the Kafka specifications used by the original service. For details about MQS specifications, see MQS Specifications.
3. Create a topic with the same configurations as the original Kafka instance, including the topic name, number of replicas, number of partitions, message aging time, and whether to enable synchronous replication and flushing.

Migration Scheme 1: Migrating the Production First

• Solution

  In this solution, first migrate the message production service to ROMA Connect so that the original Kafka does not generate new messages. After all messages in the original Kafka are consumed, migrate the message consumption service to ROMA Connect to consume new messages.

  This is a common migration solution in the industry because the operation procedure is simple. The migration process is controlled by the service side. During the entire process, messages are not out of order. However, latency may occur because there is a period when you have to wait for all data to be consumed.

  This scheme is applicable to services that require the message sequence but are insensitive to the end-to-end latency.

• Migration Process
  1. Change the Kafka connection address of the production client to the MQS connection address of the ROMA Connect instance.
  2. Restart the production service so that the producer can send new messages to the new ROMA Connect instance.
  3. Check the consumption progress of each consumer group in the original Kafka instance until all data in the original Kafka instance is consumed.
  4. Change the Kafka connection address of the consumer client to the MQS connection address of the ROMA Connect instance.
  5. Restart the consumption service so that consumers can consume messages from the ROMA Connect instance.
  6. Check whether consumers consume messages properly from the ROMA Connect instance.
  7. The migration is completed.

Migration Scheme 2: Migrating the Production Later

• Solution

  Use multiple consumers for the consumption service. Some consume messages from the original Kafka instance, and others consume messages from the ROMA Connect instance. Then, migrate the production service to the ROMA Connect instance so that all messages can be consumed in time.

  In this scheme, the consumption service may consume messages from the original Kafka and ROMA Connect at the same time in a period of time. Before the production service is migrated, the consumer service has been running on ROMA Connect. Therefore, there is no end-to-end latency problem. However, early on in the migration, data is consumed from both the original Kafka instance and ROMA Connect instance, so the messages may not be consumed in the order that they are produced.

  This scheme is suitable for services that require low latency but do not require strict message sequence.

• Migration Process
  1. Start new consumer clients, set the Kafka connection addresses to that of the ROMA Connect instance, and consume data from the ROMA Connect instance.
     

     Original consumer clients must continue running. Messages are consumed from both the original Kafka instance and ROMA Connect instance.

  2. Modify the production client and change the Kafka connection address to the MQS connection address of the ROMA Connect instance.
  3. Restart the producer client to migrate the production service to the ROMA Connect instance.
  4. After the production service is migrated, check whether the consumption service connected to the ROMA Connect instance is normal.
  5. After all data in the original Kafka is consumed, close the original consumption clients.
  6. The migration is completed.

Migrating Persistent Data

You can migrate consumed data from the original Kafka instance to the ROMA Connect instance by using the open-source tool MirrorMaker. This tool mirrors the original Kafka consumer and ROMA Connect producer and migrates data to the ROMA Connect instance.

If the topic of the original Kafka contains a single replica and the topic of the ROMA Connect instance contains three replicas, it is recommended that the storage space of the ROMA Connect instance be three times that of the original Kafka.