ALM-18031 TimelineServer Active Node Promotion Failed

Alarm Description

This alarm is generated when the TimelineHaDaemon process fails to start the TimelineServer during the promotion of an active TimelineServer instance.

This alarm is cleared when the TimelineServer on the node either starts successfully or transitions to the standby state.

This section applies to MRS 3.6.0-LTS and later versions.

Alarm Attributes

Alarm Parameters

Impact on the System

Historical records cannot be viewed on the Tez page.

Possible Causes

• The ZooKeeper service is abnormal.
• The HDFS service is abnormal.
• The floating IP address is abnormal.

Handling Procedure

Check the ZooKeeper service status.

1. Log in to FusionInsight Manager, and choose Cluster > Services. In the service list, check whether Running Status of ZooKeeper is Normal.
   • If yes, go to Step 5.
   • If no, go to Step 2.

2. Choose O&M > Alarm > Alarms. In the alarm list, check whether alarm ALM-13000 ZooKeeper Service Unavailable exists.
   • If yes, go to Step 3.
   • If no, go to Step 5.

3. Rectify the fault by performing the operations provided for ALM-13000 ZooKeeper Service Unavailable.
4. Wait 5 minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 5.

Check the HDFS service status.

5. Choose O&M > Alarm > Alarms. In the alarm list, check whether alarm ALM-14000 HDFS Service Unavailable exists.
   • If yes, go to Step 6.
   • If no, go to Step 8.

6. Rectify the fault by performing the operations provided for ALM-14000 HDFS Service Unavailable.
7. Wait 5 minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 8.

8. Choose Cluster > Services > HDFS. On the Dashboard page, check whether Safe Mode of HDFS is ON.
   • If yes, go to Step 9.
   • If no, go to Step 12.

9. Log in to the node where the HDFS client is installed as user root, configure environment variables, and authenticate the user.

   cd Client installation directory

   source bigdata_env

   kinit hdfs (Skip this step if Kerberos authentication is disabled for the cluster (the cluster is in normal mode).)

   Obtain the password of the hdfs user from the cluster administrator.

10. Run the following command to manually exit the safe mode:

    hdfs dfsadmin -safemode leave

11. Wait 5 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 12.

Check whether the floating IP address is abnormal.

12. On FusionInsight Manager, choose Cluster > Services > Yarn, click Configurations and then All Configurations, and check and record the value of TLS_FLOAT_IP.
13. On the Instances tab page of the YARN service, check and record the service IP address of the TimelineServer instance. Log in to the node where TimelineServer is installed as user root and run the following command to check whether the IP address in Step 12 is accessible:

    ping IP address in Step 12

    • If yes, go to Step 14.
    • If no, go to Step 18.

14. Run the following command to check whether the floating IP address is the TimelineServer floating IP address obtained in Step 12:

    ip addr

    • If yes, go to Step 15.
    • If no, change the floating IP address of TLS_FLOAT_IP in Step 12 to the IP address that cannot be pinged and go to Step 16.

15. Run the following command to delete the floating IP address:

    ip addr del IP address dev NIC name

16. On FusionInsight Manager, choose Cluster > Services > Yarn > Instances, select all TimelineServer instances, and choose More > Restart Instance.
17. Wait 5 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 18.

Collect fault information.

18. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
19. Expand the Service drop-down list, and select the following services for the target cluster:
    • ZooKeeper
    • HDFS
    • Mapreduce

20. Click the edit icon in the upper right corner, and select a time span starting 10 minutes before and ending 10 minutes after when the alarm was generated. Then, click Download to collect the logs.
21. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None