Partition Reassignment

Scenario

Partition reassignment is to reassign replicas of a partition to different brokers to solve the problem of unbalanced broker load.

Partition reassignment is required in the following scenarios:

• After the broker quantity is increased for an instance, the replicas of the original topic partitions are migrated to the new brokers.
• The leader partition is degraded to be a follower on a heavily loaded broker.
• The number of replicas is increased or decreased.

The DMS for Kafka console provides automatic and manual reassignment. Automatic reassignment is recommended because it ensures that leaders are evenly distributed.

Operation Impact

• Partition reassignment on topics with a large amount of data consumes a large amount of network and storage bandwidth. As a result, service requests may time out or the latency may increase. Therefore, you are advised to perform reassignment during off-peak hours.
• A throttle refers to the upper limit of the bandwidth for replication of a topic, to ensure that other topics on the instance are not affected. Note that throttles apply to replication triggered by both normal message production and partition reassignment. If the throttle is too small, normal message production may be affected, and partition reassignment may never complete.
• You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
• You cannot modify the partition quantity of topics whose reassignment tasks have started.
• Reassignment tasks cannot be manually stopped. Please wait until they complete.
• After partition reassignment, the metadata of the topic changes. If the producer does not support the retry mechanism, a few requests will fail, causing some messages to fail to be produced.
• Reassignment takes a long time if the topic has a large amount of data. You are advised to decrease the topic aging time based on the topic consumption so that historical data of the topic can be deleted in a timely manner to accelerate the migration.

Preparing for Partition Reassignment

• To reduce the amount of data to be migrated, decrease the topic aging time without affecting services and wait for messages to age. After the reassignment is complete, you can restore the aging time.
• Ensure that the target broker has sufficient disk capacity. If the remaining disk capacity of the target broker is close to the amount of data to be migrated to the broker, expand the disk capacity before the reassignment.

Auto Reassignment

1. Log in to the management console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click and choose Application > Distributed Message Service for Kafka to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose the Topics tab.
6. Reassign partitions using either of the following methods:
   • Select one or more topics and choose Reassign > Auto above the topic list.
   • In the row that contains the desired topic, choose More > Reassign > Auto.

7. Set automatic reassignment parameters.
   • In the Brokers area, select the brokers to assign the topic's partition replicas to.
   • In the Topics area, enter the number of replicas to be automatically reassigned. The number of replicas must be less than or equal to the number of brokers.
   • Specify throttle. The default value is -1, indicating that there is no throttle (recommended if the instance load is light). If a throttle is required, you are advised to set it to a value greater than or equal to the total production bandwidth of the to-be-reassigned topic multiplied by the maximum number of replicas of the to-be-reassigned topic. For details, see Calculating a Throttle.
   Figure 1 Setting automatic reassignment parameters
   

8. Click OK. The topic list is displayed.

   In the upper left corner of the topic list, click View details to view the reassignment task status on the Background Tasks page that is displayed. When the task status is Successful, reassignment has completed.

   

   • You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
   • You cannot modify the partition quantity of topics whose reassignment tasks have started.
   • Reassignment tasks cannot be manually stopped. Please wait until they complete.

Manual Reassignment

1. Log in to the management console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click and choose Application > Distributed Message Service for Kafka to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose the Topics tab.
6. Reassign partitions using either of the following methods:
   • Select the desired topic and choose Reassign > Manual above the topic list.
   • In the row that contains the desired topic, choose More > Reassign > Manual.

7. Set manual reassignment parameters.
   • In the upper right corner of the Manual dialog box, click Delete Replica or Add Replica to reduce or increase the number of replicas for each partition of the topic.
   • Under the name of the replica to be reassigned, click the broker name or and select the target broker to migrate the replica to. Assign replicas of the same partition to different brokers.
   • Specify throttle. The default value is -1, indicating that there is no throttle (recommended if the instance load is light). If a throttle is required, you are advised to set it to a value greater than or equal to the total production bandwidth of the to-be-reassigned topic multiplied by the maximum number of replicas of the to-be-reassigned topic. For details, see Calculating a Throttle.
   Figure 2 Setting manual reassignment parameters
   

8. Click OK. The topic list is displayed.

   In the upper left corner of the topic list, click View details to view the reassignment task status on the Background Tasks page that is displayed. When the task status is Successful, reassignment has completed.

   

   • You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
   • You cannot modify the partition quantity of topics whose reassignment tasks have started.
   • Reassignment tasks cannot be manually stopped. Please wait until they complete.

Calculating a Throttle

Throttles are affected by the execution duration of the reassignment, leader/follower distribution of partition replicas, and message production rate.

• A throttle limits the replication traffic of all partitions in a broker.
• Replicas added after the assignment are regarded as followers, and existing replicas are regarded as leaders. Throttles on leaders and followers are separated.
• Throttles do not distinguish between replication caused by normal message production and that caused by partition reassignment. Therefore, the traffic generated in both cases is throttled.

Assume that the partition reassignment task needs to be completed within 200s and each replica has 100 MB data. Calculate the throttle in the following scenarios:

Scenario 1: Topic 1 has two partitions and two replicas, and Topic 2 has one partition and one replica. All leader replicas are on the same broker. One replica needs to be added for Topic 1 and Topic 2 respectively.

Figure 3 Reassignment scenario 1

As shown in Figure 3, three replicas fetch data from Broker 0. Each replica on Broker 0 has 100 MB data. Broker 0 has only leader replicas, and Broker 1 and Broker 2 have only follower replicas.

• Bandwidth required by Broker 0 to complete partition reassignment within 200s = (100 MB + 100 MB + 100 MB)/200s = 1.5 MB/s
• Bandwidth required by Broker 1 to complete partition reassignment within 200s = 100 MB/200s = 0.5 MB/s
• Bandwidth required by Broker 2 to complete partition reassignment within 200s = (100 MB + 100 MB)/200s = 1 MB/s

In conclusion, to complete the partition reassignment task within 200s, set the throttle to a value greater than or equal to 1.5 MB/s.

Scenario 2: Topic 1 has two partitions and one replica, and Topic 2 has two partitions and one replica. Leader replicas are on different brokers. One replica needs to be added for Topic 1 and Topic 2 respectively.

Figure 4 Reassignment scenario 2