ALM-38000 Kafka Service Unavailable

Alarm Description

The system checks the Kafka service status every 30 seconds. This alarm is generated when the Kafka service is unavailable.

This alarm is cleared when the Kafka service recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

The cluster cannot provide the Kafka service, and users cannot perform new Kafka tasks.

Possible Causes

• The KrbServer service is faulty (for non-normal clusters).
• The ZooKeeper service is abnormal or does not respond.
• The Broker instances in the Kafka cluster are abnormal.

Handling Procedure

Check the status of the KrbServer service. (Skip this step if the normal mode is used.)

1. Log in to FusionInsight Manager and choose Cluster > Services > KrbServer.
2. Check whether the running status of the KrbServer service is Normal.
   • If yes, go to Step 5.
   • If no, go to Step 3.

3. Rectify the fault by following the steps provided in ALM-25500 KrbServer Service Unavailable.
4. Perform Step 2 again.

Check the status of the ZooKeeper cluster.

5. Check whether the running status of the ZooKeeper service is Normal.
   • If yes, go to Step 8.
   • If no, go to Step 6.

6. If ZooKeeper service is stopped, start it, else rectify the fault by following the steps provided in ALM-13000 ZooKeeper Service Unavailable.
7. Perform Step 5 again.

Check the Broker status.

8. Choose Cluster > Services > JobGateway and click Instances. The Kafka instance page is displayed.
9. Check whether all instances in Roles are running properly.
   • If yes, go to Step 11.
   • If no, go to Step 10.

10. Select all Broker instances, choose More > Restart Instance, and check whether the instances restart successfully.
    

    During the restart of the Broker instance, if the current Topic is a single copy and is on the current Broker node, the Kafka service will be interrupted. Otherwise, the Kafka service will not be affected.

    • If yes, go to Step 11.
    • If no, go to Step 13.

11. Choose Cluster > Services > Kafka and check whether the service status is Normal.
    • If yes, go to Step 12.
    • If no, go to Step 13.

12. Wait for 30 seconds and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 13.

Collect fault information.

13. On the FusionInsight Manager portal, choose O&M > Log > Download.
14. Select Kafka in the required cluster from the Service drop-down list.
15. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
16. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None