Reassigning Kafka Partitions

Scenario

Partition reassignment is to reassign replicas of a partition to different brokers to solve the problem of unbalanced broker load.

Partition reassignment is required in the following scenarios:

• After the broker quantity is increased for an instance, the replicas of the original topic partitions are migrated to the new brokers.
• The leader partition is degraded to be a follower on a heavily loaded broker.
• The number of replicas is increased or decreased.

The DMS for Kafka console provides automatic and manual reassignment. Automatic reassignment is recommended because it ensures that leaders are evenly distributed.

Operation Impact

• Partition reassignment on topics with a large amount of data consumes a large amount of network and storage bandwidth. As a result, service requests may time out or the latency may increase. Therefore, you are advised to perform reassignment during off-peak hours. Compare the current instance load based on the instance specifications to decide whether the remaining instance capacity can support partition reassignment. Do not reassign partitions when there is insufficient bandwidth or when the CPU usage is greater than 90%.
• A throttle refers to the upper limit of the bandwidth for replication of a topic, to ensure that other topics on the instance are not affected. Note that throttles apply to replication triggered by both normal message production and partition reassignment. If the throttle is too small, normal message production may be affected, and partition reassignment may never complete.
• You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
• You cannot modify the partition quantity of topics whose reassignment tasks have started.
• Reassignment tasks cannot be manually stopped. Please wait until they complete.
• After partition reassignment, the metadata of the topic changes. If the producer does not support the retry mechanism, a few requests will fail, causing some messages to fail to be produced.
• Reassignment takes a long time if the topic has a large amount of data. You are advised to decrease the topic aging time based on the topic consumption so that historical data of the topic can be deleted in a timely manner to accelerate the migration.

Preparing for Partition Reassignment

• To reduce the amount of data to be migrated, decrease the topic aging time without affecting services and wait for messages to age. After the reassignment is complete, you can restore the aging time.
• Ensure that the target broker has sufficient disk capacity. If the remaining disk capacity of the target broker is close to the amount of data to be migrated to the broker, expand the disk capacity before the reassignment.

Auto Reassignment

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click and choose Application > Distributed Message Service for Kafka to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose the Topics tab.
6. Reassign partitions using either of the following methods:
   • Select one or more topics and choose Reassign > Auto above the topic list.
   • In the row that contains the desired topic, choose More > Reassign > Auto.

7. Set automatic reassignment parameters.
   • In the Brokers area, select the brokers to assign the topic's partition replicas to.
   • In the Topics area, enter the number of replicas to be automatically reassigned. The number of replicas must be less than or equal to the number of brokers.
   • Specify throttle. The default value is -1, indicating that there is no throttle (recommended if the instance load is light). If a throttle is required, you are advised to set it to a value greater than or equal to the total production bandwidth of the to-be-reassigned topic multiplied by the maximum number of replicas of the to-be-reassigned topic. For details, see Calculating a Throttle.
   • For Execute, specify when to execute the reassignment. Now means to execute it immediately. As scheduled means to execute it at the scheduled time.
   Figure 1 Setting automatic reassignment parameters
   

8. (Optional) Click Calculate. Time Required indicates how long automatic balancing will take.
9. Click OK.

   The following table lists how to check whether reassignment is complete (scheduled and non-scheduled tasks):

   Table 1 Checking the reassignment result

   Task Type	Reassignment Result
   Background tasks	In the upper left corner of the topic list, click View details and the Background Tasks > Background tasks page is displayed. The reassignment task is complete when it is in the Successful state.
   Scheduled tasks	The Background Tasks > Scheduled tasks page is displayed. This page only shows whether scheduled tasks start to execute instead of whether they are successful. When the task status is Pending, reassignment has not been executed. When the task status is Successful, reassignment has started. Click Background tasks tab page. When the task status is Successful, reassignment has completed.

   

   • You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
   • You cannot modify the partition quantity of topics whose reassignment tasks have started.
   • Reassignment tasks cannot be manually stopped. Please wait until they complete.
   • If a scheduled partition reassignment task has been configured, no other reassignment can be executed until this existing task is executed.

Manual Reassignment

1. Log in to the console.
2. Click in the upper left corner to select a region.
   

   Select the region where your Kafka instance is located.

3. Click and choose Application > Distributed Message Service for Kafka to open the console of DMS for Kafka.
4. Click the desired Kafka instance to view the instance details.
5. In the navigation pane, choose the Topics tab.
6. Reassign partitions using either of the following methods:
   • Select a topic and choose Reassign > Manual above the topic list. Manual reassignment does not support batch operations.
   • In the row that contains the desired topic, choose More > Reassign > Manual.

7. Set manual reassignment parameters.
   • In the upper right corner of the Manual dialog box, click Delete Replica or Add Replica to reduce or increase the number of replicas for each partition of the topic.
   • Under the name of the replica to be reassigned, click the broker name or and select the target broker to migrate the replica to. Assign replicas of the same partition to different brokers.
   • Specify throttle. The default value is -1, indicating that there is no throttle (recommended if the instance load is light). If a throttle is required, you are advised to set it to a value greater than or equal to the total production bandwidth of the to-be-reassigned topic multiplied by the maximum number of replicas of the to-be-reassigned topic. For details, see Calculating a Throttle.
   • For Execute, specify when to execute the reassignment. Now means to execute it immediately. As scheduled means to execute it at the scheduled time.
   Figure 2 Setting manual reassignment parameters
   

8. (Optional) Click Calculate. Time Required indicates how long manual balancing will take.
9. Click OK.

   The following table lists how to check whether reassignment is complete (scheduled and non-scheduled tasks):

   Table 2 Checking the reassignment result

   Task Type	Reassignment Result
   Background tasks	In the upper left corner of the topic list, click View details and the Background Tasks > Background tasks page is displayed. The reassignment task is complete when it is in the Successful state.
   Scheduled tasks	The Background Tasks > Scheduled tasks page is displayed. This page only shows whether scheduled tasks start to execute instead of whether they are successful. When the task status is Pending, reassignment has not been executed. When the task status is Successful, reassignment has started. Click Background tasks tab page. When the task status is Successful, reassignment has completed.

   

   • You cannot delete topics whose reassignment tasks have started. Otherwise, the tasks will never complete.
   • You cannot modify the partition quantity of topics whose reassignment tasks have started.
   • Reassignment tasks cannot be manually stopped. Please wait until they complete.
   • If a scheduled partition reassignment task has been configured, no other reassignment can be executed until this existing task is executed.

Re-scheduling Partition Reassignment

1. On the Scheduled tasks tab page on the Background Tasks page, click the drop-down box in the upper left corner, select a time period, enter the desired topic name in the search box, and press Enter.
   Figure 3 Querying a scheduled reassignment task
   

2. In the row that contains the desired task, click Modify.
3. In the Change Schedule dialog box, change the schedule or cancel the scheduled task.
   • To change the schedule, select a time and click OK.
   • To cancel the task, select Cancel (as shown in Figure 4) and click OK.
     Figure 4 Canceling a scheduled reassignment task
     

Calculating a Throttle

Throttles are affected by the execution duration of the reassignment, leader/follower distribution of partition replicas, and message production rate.

• A throttle limits the replication traffic of all partitions in a broker.
• Replicas added after the assignment are regarded as followers, and existing replicas are regarded as leaders. Throttles on leaders and followers are separated.
• Throttles do not distinguish between replication caused by normal message production and that caused by partition reassignment. Therefore, the traffic generated in both cases is throttled.

Assume that the partition reassignment task needs to be completed within 200s and each replica has 100 MB data. Calculate the throttle in the following scenarios:

Scenario 1: Topic 1 has two partitions and two replicas, and Topic 2 has one partition and one replica. All leader replicas are on the same broker. One replica needs to be added for Topic 1 and Topic 2 respectively.

Figure 5 Reassignment scenario 1

As shown in Figure 5, three replicas fetch data from Broker 0. Each replica on Broker 0 has 100 MB data. Broker 0 has only leader replicas, and Broker 1 and Broker 2 have only follower replicas.

• Bandwidth required by Broker 0 to complete partition reassignment within 200s = (100 MB + 100 MB + 100 MB)/200s = 1.5 MB/s
• Bandwidth required by Broker 1 to complete partition reassignment within 200s = 100 MB/200s = 0.5 MB/s
• Bandwidth required by Broker 2 to complete partition reassignment within 200s = (100 MB + 100 MB)/200s = 1 MB/s

In conclusion, to complete the partition reassignment task within 200s, set the throttle to a value greater than or equal to 1.5 MB/s.

Scenario 2: Topic 1 has two partitions and one replica, and Topic 2 has two partitions and one replica. Leader replicas are on different brokers. One replica needs to be added for Topic 1 and Topic 2 respectively.

Figure 6 Reassignment scenario 2