ALM-38001 Insufficient Kafka Disk Capacity

Description

The system checks the Kafka disk usage every 60 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold. This alarm is generated when the disk usage is greater than the threshold.

You can change the threshold in O&M > Alarm > Thresholds. Under the service list, choose Kafka > Disk > Broker Disk Usage (Broker) and change the threshold.

When the Trigger Count is 1, this alarm is cleared when the Kafka disk usage is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the Kafka disk usage is less than or equal to 90% of the threshold.

Attribute

Parameters

Impact on the System

Kafka data write operations are affected.

Possible Causes

• The configuration (such as number and size) of the disks for storing Kafka data cannot meet the requirement of the current service traffic, due to which the disk usage reaches the upper limit.
• Data retention time is too long, due to which the data disk usage reaches the upper limit.
• The service plan does not distribute data evenly, due to which the usage of some disks reaches the upper limit.

Procedure

Check the disk configuration of Kafka data.

1. On the FusionInsight Manager portal and click O&M > Alarm > Alarms.
2. In the alarm list, locate the alarm and obtain HostName from Location.
3. Click Cluster > Name of the desired cluster > Hosts.
4. In the host list, click the host name obtained in 2.
5. Check whether the Disk area contains the partition name in the alarm.
   • If yes, go to 6.
   • If no, manually clear the alarm and no further operation is required.

6. Check whether the disk partition usage contained in the alarm reaches 100% in the Disk area.
   • If yes, handle the alarm by following the instructions in Related Information.
   • If no, go to 7.

Check the Kafka data storage duration.

7. Choose Cluster > Name of the desired cluster > Services > Kafka > Configurations.
8. Check whether the value of parameter disk.adapter.enable is set to true.
   • If yes, go to 10.
   • If no, go to 9.

9. Set the value of disk.adapter.enable to true. Check whether the value of adapter.topic.min.retention.hours is properly set.
   • If yes, go to 10.
   • If no, adjust the data retention period based on service requirements.
   

   If the disk auto-adaptation function is enabled, some historical data of specified topics is deleted. If the retention period of some topics cannot be adjusted, click All Configurations and add the topics to the value of the disk.adapter.topic.blacklist parameter.

10. Wait 10 minutes and check whether the usage of faulty disks reduces.
    • If yes, wait until the alarm is cleared.
    • If no, go to 11.

Check the Kafka data plan.

11. In the Instance area, click Broker. In the Real Time area of Broker, Click the drop-down menu in the Chart area and choose Customize to customize monitoring items.
12. In the dialog box, select Disk > Broker Disk Usage and click OK.

    The Kafka disk usage information is displayed.

    Figure 1 Broker Disk Usage
    

13. View the information in 12 to check whether there is only the disk parathion for which the alarm is generated in 2.
    • If yes, go to 14.
    • If no, go to 15.

14. Perform disk planning and mount a new disk again. Go to the Instance Configurations page of the node for which the alarm is generated, modify log.dirs, add other disk directories, and restart the Kafka instance.
    

    During the restart of the Broker instance, if the current Topic is a single copy and is on the current Broker node, the Kafka service will be interrupted. Otherwise, the Kafka service will not be affected.

15. Determine whether to shorten the data retention time configured on Kafka based on service requirements and service traffic.
    • If yes, go to 16.
    • If no, go to 17.

16. Log in to FusionInsight Manager, select Cluster > Name of the desired cluster > Services > Kafka > Configurations, and click All Configurations. In the search box on the right, enter log.retention.hours. The value of the parameter indicates the default data retention time of the topic. You can change the value to a smaller one.
    

    • For a topic whose data retention time is configured alone, the modification of the data retention time on the Kafka Service Configuration page does not take effect.
    • To modify the data retention time for a topic, use the Kafka client command-line interface (CLI) to configure the topic.

      Example: kafka-topics.sh --zookeeper "ZooKeeper IP address:2181/kafka" --alter --topic "Topic bane" --config retention.ms= "retention time"

17. Check whether the usage of some disks reaches the upper limit due to unreasonable configuration of the partitions of some topics. For example, the number of partitions configured for a topic with large data volume is smaller than the number of disks. In this case, the data is not evenly allocated to disks.
    

    If you do not know which topics have a large amount of service data, perform the following steps:

    1. Log in to an instance node based on the host node information obtained in 2.
    2. Go to the data directory (directory specified by log.dirs before the modification in 14).
    3. Run the following command to check whether there is topic with partition that use large disk space.

       du -h --max-depth=1 ./

    • If yes, go to 18.
    • If no, go to 19.

18. In the Kafka client CLI, run the following command to perform partition capacity expansion for the topic:

    kafka-topics.sh --zookeeper "ZooKeeper IP address:2181/kafka" --alter --topic "Topic name" --partitions="New number of partitions"

    

    • You are advised to set the new number of partitions to a multiple of the number of Kafka data disks.
    • The step may not quickly clear the alarm, and you need to modify the data retention time in 11 to gradually balance data allocation.

19. Determine whether to perform capacity expansion.
    

    You are advised to perform capacity expansion for Kafka when the current disk usage exceeds 80%.

    • If yes, go to 20.
    • If no, go to 21.

20. Expand the disk capacity and check whether the alarm is cleared after capacity expansion.
    • If yes, no further action is required.
    • If no, go to 22.

21. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 22.

Collect fault information.

22. On the FusionInsight Manager portal, choose O&M > Log > Download.
23. Select Kafka in the required cluster from the Service drop-down list.
24. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
25. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

1. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Kafka > Instance, stop the Broker instance whose status is Restoring, record the management IP address of the node where the Broker instance is located, and record broker.id. The value can be obtained by using the following method: Click the role name. On the Configurations page, select All Configurations, and search for the broker.id parameter.
2. Log in to the recorded management IP address as user root, and run the df -lh command to view the mounted directory whose disk usage is 100%, for example, ${BIGDATA_DATA_HOME}/kafka/data1.
3. Go to the directory, run the du -sh * command to view the size of each file in the directory,check whether files other than kafka-logs exist, and determine whether these files can be deleted or migrated.
   • If yes, go to 8.
   • If no, go to 4.

4. Go to the kafka-logs directory, run the du -sh * command, select a partition folder to be moved. The naming rule is Topic name-Partition ID. Record the topic and partition.
5. Modify the recovery-point-offset-checkpoint and replication-offset-checkpoint files in the kafka-logs directory in the same way.
   1. Decrease the number in the second line in the file. (To remove multiple directories, the number deducted is equal to the number of files to be removed.)
   2. Delete the line of the to-be-removed partition. (The line structure is "Topic name Partition ID Offset". Save the data before deletion. Subsequently, the content must be added to the file of the same name in the destination directory.)

6. Modify the recovery-point-offset-checkpoint and replication-offset-checkpoint files in the destination data directory. For example, ${BIGDATA_DATA_HOME}/kafka/data2/kafka-logs in the same way.
   • Increase the number in the second line in the file. (To move multiple directories, the number added is equal to the number of files to be moved.)
   • Add the to-be moved partition to the end of the file. (The line structure is "Topic name Partition ID Offset". You can copy the line data saved in 5.)

7. Move the partition to the destination directory. After the partition is moved, run the chown omm:wheel -R Partition directory command to modify the directory owner group for the partition.
8. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Kafka > Instance to start the Broker instance.
9. Wait for 5 to 10 minutes and check whether the health status of the Broker instance is Normal.
   • If yes, resolve the disk capacity insufficiency problem according to the handling method of "ALM-38001 Insufficient Kafka Disk Space" after the alarm is cleared.
   • If no, contact the O&M personnel.