ALM-19022 HBase Hotspot Detection Is Unavailable

Alarm Description

When the MetricController instance is installed for HBase, the alarm module checks the health status of the active HBase MetricController instance every 120 seconds. This alarm is generated when the active HBase MetricController instance does not exist or is unavailable and the hotspot detection function is unavailable.

This alarm is cleared when the active HBase MetricController instance recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

The HBase hotspot detection function is unavailable. Services are not affected. However, if request/data skew occurs, the system cannot report alarms and automatically recovers from hotspotting. Service requests may cause node overload, slow response, and request timeout.

Possible Causes

• The ZooKeeper service is abnormal.
• The HBase service is abnormal.
• In the current HBase service, the MetricController instance on the same node as the active HMaster instance is not started.
• The network is abnormal.

Handling Procedure

Check the ZooKeeper service status.

1. In the service list on FusionInsight Manager, check whether Running Status of ZooKeeper is Normal.
   • If yes, go to 5.
   • If no, go to 2.

2. In the alarm list, check whether ALM-13000 ZooKeeper Service Unavailable exists.
   • If yes, go to 3.
   • If no, go to 5.

3. Rectify the fault by performing the operations provided for ALM-13000 ZooKeeper Service Unavailable.
4. Wait for several minutes and check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
   • If yes, no further action is required.
   • If no, go to 5.

Check the HBase service status.

5. In the service list on FusionInsight Manager, check whether Running Status of HBase is Normal.
   • If yes, go to 9.
   • If no, go to 6.

6. In the alarm list, check whether the alarm ALM-19000 HBase Service Unavailable exists.
   • If yes, go to 7.
   • If no, go to 9.

7. Rectify the fault by following the steps provided for ALM-19000 HBase Service Unavailable.
8. Wait for several minutes and check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Check whether the MetricController instance deployed on the same node as the active HMaster instance is started.

9. On FusionInsight Manager, choose Cluster > Service > HBase, and click Instances to check whether the MetricController(Active) instance exists.
   • If yes, go to 12.
   • If no, go to 10.

10. Select the MetricController instance whose management IP address is the same as that of the active HMaster instance, and click Start Instance.
11. After the MetricController instance is restarted, check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to 12.

Check the network connectivity between the started MetricController instances and the active HMaster node.

12. Log in to the node where the active HMaser instance is deployed and run ping IP address of the node where the standby MetricController instance is deployed to check whether the network connection between the started MetricController instances and the host where the active HMaster instance is deployed is normal.
    • If yes, go to 15.
    • If no, go to 13.

13. Contact the network administrator to restore the network.
14. After the network recovers, check whether the alarm HBase Hotspot Detection Is Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to 15.

Collect fault information.

15. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
16. Expand the Service drop-down list, and select HBase for the target cluster.
17. In the Host area, select the host where the HMaster instance is deployed.
18. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
19. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.