ALM-23001 Loader Service Unavailable

Alarm Description

The system checks the Loader service availability every 60 seconds. This alarm is generated if the Loader service is unavailable and is cleared after the Loader service recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

Data loading, import, and conversion are unavailable.

Possible Causes

• The services that Loader depends on are abnormal.
  • The ZooKeeper service is abnormal.
  • The HDFS service is abnormal.
  • The DBService service is abnormal.
  • The Yarn service is abnormal.
  • MapReduce is abnormal.
• The network is faulty. Loader cannot communicate with its dependent services.
• Loader is running improperly.

Handling Procedure

Check the ZooKeeper service status.

1. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper to check whether ZooKeeper is running properly.
   • If yes, go to 3.
   • If no, go to 2.

2. Choose More > Restart Service to restart ZooKeeper. After ZooKeeper starts, check whether the "Loader Service Unavailable" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 3.

3. In the alarm list on the FusionInsight Manager, check whether the alarm "Process Fault" is generated.
   • If yes, go to 4.
   • If no, go to 7.

4. In Location of ALM-12007 Process Fault, check whether the service name is ZooKeeper.
   • If yes, go to 5.
   • If no, go to 7.

5. Rectify the fault by following steps provided in ALM-12007 Process Fault.
6. In the alarm list, check whether this alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Check the HDFS service status.

7. In the alarm list on FusionInsight Manager, check whether an alarm is generated indicating that the HDFS service is unavailable.
   • If yes, go to 8.
   • If no, go to 10.

8. Rectify the fault by performing the operations provided for ALM-14000 HDFS Service Unavailable.
9. In the alarm list, check whether this alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 10.

Check the DBService status.

10. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > DBService to check whether DBService is running properly.
    • If yes, go to 12.
    • If no, go to 11.

11. Choose More > Restart Service to restart DBService. After DBService starts, check whether the "Loader Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 12.

Check the MapReduce status.

12. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Mapreduce to check whether MapReduce is running properly.
    • If yes, go to 16.
    • If no, go to 13.

13. Choose More > Restart Service to restart MapReduce. After MapReduce starts, check whether the "Loader Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 16.

Check the Yarn service status.

14. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Yarn to check whether Yarn is running properly.
    • If yes, go to 16.
    • If no, go to 15.

15. Choose More > Restart Service to restart Yarn. After Yarn starts, check whether the "Loader Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 16.

16. In the alarm list on FusionInsight Manager, check whether an alarm is generated indicating that the Yarn service is unavailable.
    • If yes, go to 17.
    • If no, go to 19.

17. Rectify the fault by performing the operations provided for ALM-18000 Yarn Service Unavailable.
18. In the alarm list, check whether this alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 19.

Check the network connections between Loader and its dependent components.

19. On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose Services > Loader.
20. Click Instance. The LoaderServer instance list is displayed.
21. Record the management IP address in the LoaderServer(Active) row.
22. Log in to the host where LoaderServer resides as user omm using the IP address obtained in 21.

23. Run the ping command to check whether the network connection between the hosts where the LoaderServer instances reside and the dependent components is normal. (The dependent components include ZooKeeper, DBService, HDFS, MapReduce, and Yarn. The method to obtain the IP addresses of the dependent components is the same as that used to obtain the IP addresses of the active LoaderServer instances.)
    • If yes, go to 26.
    • If no, go to 24.

24. Contact the network administrator to restore the network.
25. In the alarm list, check whether this alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 26.

Collect fault information.

26. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
27. Expand the Service drop-down list, and select the following services for the target cluster:
    • Zookeeper
    • HDFS
    • DBService
    • Yarn
    • MapReduce
    • Loader

28. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
29. On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose Services > Loader.
30. Choose More > Restart Service and click OK.

31. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 32.

32. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.