ALM-38001 Insufficient Kafka Disk Space

Alarm Description

The system checks the Kafka disk usage every 60 seconds and compares the actual disk usage with the threshold. The disk usage has a default threshold. This alarm is generated if the disk usage exceeds the threshold.

To change the threshold, choose O&M > Alarm > Thresholds. In the service list, choose Kafka > Disk > Broker Disk Usage (Broker).

If the Trigger Count is 1, this alarm is cleared when the usage of the Kafka disk is less than or equal to the threshold. If the Trigger Count is greater than 1, this alarm is cleared when the disk usage is less than or equal to 80% of the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

Kafka data write operations fail.

Possible Causes

• The Kafka disk configurations (such as disk count and disk size) are insufficient for the data volume.
• The data retention period is long and historical data occupies large space.
• Services are improperly planned. As a result, data is unevenly distributed and some disks are full.

Handling Procedure

Check the disk configuration of Kafka data.

1. Log in to FusionInsight Manager and choose O&M > Alarm > Alarms.
2. In the alarm list, click the alarm and obtain the host name from the Location area.
3. Choose Cluster > Name of the desired cluster > Hosts.
4. On the Hosts page, click the host name obtained in 2.
5. Check whether the Disk area contains the disk partition name in the alarm.
   • If yes, go to 6.
   • If no, manually clear the alarm and no further action is required.

6. In the Disk area, check whether the usage of the alarmed partition has reached 100%.
   • If yes, handle the alarm by following the instructions in Related Information.
   • If no, go to 7.

Check the Kafka data storage duration.

7. Choose Cluster > Name of the desired cluster > Services > Kafka > Configurations > All Configurations.
8. Check whether the value of disk.adapter.enable is true.
   • If yes, go to 10.
   • If no, go to 9.

9. Set disk.adapter.enable to true. Check whether the value of adapter.topic.min.retention.hours is properly set.
   • If yes, go to 10.
   • If no, adjust the data retention period based on service requirements.
   

   If the disk adaptation function is enabled, historical data of topics may be deleted. If the retention period of some topics cannot be adjusted, click All Configurations and add the topics to the value of the disk.adapter.topic.blacklist parameter.

10. Wait for 10 minutes and check whether the usage of the faulty disk decreases.
    • If yes, wait until the alarm is cleared.
    • If no, go to 11.

Check the Kafka data plan.

11. In the Instance area, click Broker. Click the drop-down menu in the Chart area and choose Customize to customize monitoring items.
12. In the displayed dialog box, choose Disk, select Broker Disk Usage, and click OK.

    The Kafka disk usage information is displayed.

13. View the information in 12 to check whether there is only the disk partition for which the alarm is generated in 2.
    • If yes, go to 14.
    • If no, go to 15.

14. Perform disk planning and mount a new disk again. Go to the Instance Configurations page of the node for which the alarm is generated, modify log.dirs, add other disk directories, and restart the Kafka instance.
15. Determine whether to shorten the data retention time configured on Kafka based on service requirements and service traffic.
    • If yes, go to 16.
    • If the alarm fails to be cleared, go to 17.

16. Log in to FusionInsight Manager, select Cluster > Name of the desired cluster > Services > Kafka > Configurations, and click All Configurations. In the search box on the right, enter log.retention.hours. The value of the parameter indicates the default data retention time of the topic. You can change the value to a smaller one.
    

    • For a topic whose data retention time is configured alone, the modification of the data retention time on the Kafka service configuration page does not take effect.

    • To modify the data retention time for a topic, use the Kafka client command-line interface (CLI).

      For example: kafka-topics.sh --zookeeper "ZooKeeper address :2181 /kafka" --alter --topic" Topic name" --config retention.ms="Storage duration"

17. Check whether partitions are properly configured for topics. For example, if the number of partitions for a topic with a large data volume is smaller than the number of disks, data may be unevenly distributed to the disks and the usage of some disks will reach the upper limit.
    

    If you do not know which topics have a large amount of service data, perform the following steps:

    1. Log in to an instance node based on the host node information obtained in 2.
    2. Go to the data directory (directory specified by log.dirs before the modification in 14).
    3. Check whether there is a topic with partition that uses large disk space.
    • If yes, go to 18.
    • If no, go to 19.

18. On the Kafka client, add partitions to the topics.

    kafka-topics.sh --zookeeper "ZooKeeperaddress:2181 /kafka" --alter --topic"Topic name" --partitions="Number of new partitions"

    

    • You are advised to set the new number of partitions to a multiple of the number of Kafka data disks.
    • The step may not quickly clear the alarm, and you need to modify the data retention time in 11 to gradually balance data allocation.

19. Determine whether to perform capacity expansion.
    

    You are advised to perform capacity expansion for Kafka when the current disk usage exceeds 80%..

    • If yes, go to 20.
    • If no, go to 21.

20. Expand the disk capacity and check whether the alarm is cleared after capacity expansion.
    • If yes, no further action is required.
    • If no, go to 22.

21. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If yes, go to 22.

Collect fault information.

22. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
23. Expand the Service drop-down list, and select Kafka for the target cluster.
24. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
25. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

1. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Kafka > Instances, stop the Broker instance in the Restoring state, and record the management IP address of the node where the instance is deployed and the corresponding broker.id. You can click the role name to view the value, on the Configurations page, select All Configurations and search for the broker.id parameter.
2. Log in to the management IP address recorded as the root user and run the df -lh command to view the mount directory whose disk usage is 100%, for example, ${BIGDATA_DATA_HOME}/kafka/data1.
3. Go to the directory, run the du -sh * command to view the size of each file in the directory. Check whether there are files in addition to the files in the kafka-logs directory, and determine whether these files can be deleted or migrated.
   • If yes, delete or migrate related data and go to 8.
   • If no, go to 4.

4. Go to the kafka-logs directory, run the du -sh * command, select a partition folder to be moved. The naming rule is Topic name-Partition ID. Record the topic and partition.
5. Modify the recovery-point-offset-checkpoint and replication-offset-checkpoint files in the kafka-logs directory in the same way.
   1. Decrease the number in the second line in the file. (To remove multiple directories, the number deducted is equal to the number of files to be removed.)
   2. Delete the line where the partition to be removed is. (The line is in "Topic name Partition ID Offset" format. Save the data before deletion. The content must be added to the file of the same name in the destination directory.)

6. Modify the recovery-point-offset-checkpoint and replication-offset-checkpoint files in the destination data directory (for example, ${BIGDATA_DATA_HOME}/kafka/data2/kafka-logs) in the same way.
   • Increase the number in the second line in the file. (To move multiple directories, the number added must be equal to the number of files to be moved.)
   • Add the partition to the end of the file. (The line structure is "Topic name Partition ID Offset". You can copy the line data saved in 5.)

7. Move the partition to the destination directory. After the partition is moved, run the chown omm:wheel -R Partition directory command to modify the directory owner group for the partition.
8. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Kafka > Instances to start the stopped Broker instance.
9. Wait for 5 to 10 minutes and check whether the health status of the Broker instance is Good.
   • If yes, rectify the fault by following the handling suggestion of ALM-38001 Insufficient Kafka Disk Capacity.
   • If no, contact O&M engineers.