ALM-12007 Process Fault

Alarm Description

The process health check module checks the process status every 5 seconds. This alarm is generated when the process health check module detects that the process connection is faulty for three consecutive times.

This alarm is cleared when the process can be connected.

Alarm Attributes

Alarm Parameters

Impact on the System

The impact varies depending on the instance that is faulty.

For example, if an HDFS instance is faulty, the impacts are as follows:

• If a DataNode instance is faulty, read and write operations cannot be performed on data blocks stored on the DataNode, which may cause data loss or unavailability. However, data in HDFS is redundant. Therefore, the client can access data from other DataNodes.
• If an HttpFS instance is faulty, the client cannot access files in HDFS over HTTP. However, the client can use other methods (such as shell commands) to access files in HDFS.
• If a JournalNode instance is faulty, namespaces and data logs cannot be stored to disks, which may cause data loss or unavailability. However, HDFS stores backups on other JournalNodes. Therefore, the faulty JournalNode can be recovered and data can be rebalanced.
• If a NameNode deployed in active/standby mode is faulty, an active/standby switchover occurs. If only one NameNode is deployed, the client cannot read or write any HDFS data. On MRS, NameNodes must be deployed in two-node mode.
• If a Router instance is faulty, the client cannot access data on the router. However, the client can use other Routers or directly access data on the backend NameNode.
• If a ZKFC instance is faulty, the NameNode does not continuously and automatically fail over. As a result, data cannot be read from or write to HDFS by the client. In this case, you need to enable automatic failover on other available ZKFC instances to restore the HDFS cluster.

Possible Causes

• The instance process is abnormal.
• The disk space is insufficient.

If a large number of process fault alarms exist in a time segment, files in the installation directory may be deleted mistakenly or permission on the directory may be modified.

Handling Procedure

Check whether the instance process is abnormal.

1. In the FusionInsight Manager portal, click O&M > Alarm > Alarms, click in the row where the alarm is located, and click the host name to view the host address for which the alarm is generated
2. On the Alarms page, check whether the ALM-12006 Node Fault is generated.
   • If yes, go to Step 3.
   • If no, go to Step 4.

3. Handle the alarm according to ALM-12006 Node Fault.
4. Log in to the host for which the alarm is generated as user root. Check whether the installation directory user, user group, and permission of the alarm role are correct. The user, user group, and the permission must be omm:ficommon 750.

   For example, the NameNode installation directory is ${BIGDATA_HOME}/FusionInsight_Current/1_8_NameNode/etc.

   • If yes, go to Step 6.
   • If no, go to Step 5.

5. Run the following command to set the permission to 750 and User:Group to omm:ficommon:

   chmod 750 <folder_name>

   chown omm:ficommon <folder_name>

6. Wait for 5 minutes. In the alarm list, check whether ALM-12007 Process Fault is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

7. Log in to the active OMS node as user root and run the following command to view the configurations.xml file. In the preceding command, "Service name" is the service name queried in Step 1.

   vi $BIGDATA_HOME/components/current/Service name/configurations.xml

   Search for the keyword healthMonitor.properties, find the health check configuration item corresponding to the alarm reporting instance, and record the interface or script path specified by monitor.info, as shown in the following figure.

   Check the logs recorded in the interface or script and rectify the fault.

   

8. Wait for 5 minutes. In the alarm list, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 9.

9. Check whether the required disk is mounted on the faulty node normally.
   1. Log in to the node where the alarm was generated as the root user and run the following command to check whether there is a disk whose TYPE is part and MOUNTPOINT is empty.

      lsblk

      • If yes, the disk is not mounted. Go to 9.b.
      • If no, the disk is mounted. Go to Step 11.

      If information similar to the following is displayed, disk vdb1 is not mounted.

      

   2. Initialize the disk that fails to be mounted and mount it again. For details, see Extending Disk Partitions and File Systems.
   3. Log in to the node for which the alarm was generated as the root user and run the following command to check and record the UUID of the mounted disk:

      blkid

   4. Run the following command to check whether there is the UUID of the mounted disk:

      vi /etc/fstab

      • If yes, go to 9.f.
      • If no, go to 9.e.

      The command output is shown in the following figure.

      

   5. Add the mounted disk information to the /etc/fstab file. You can obtain the information in any of the following ways:
      • On the node where the alarm was generated, run the following command to view historical snapshot files and check the mounting information of the offline disk:

        cat /usr/local/diskmgt/ini/fstab.xxx

      • Obtain mounting information from the /etc/fstab file on a normal node in the current cluster.
      • Contact O&M personnel.
   6. Restart the faulty instance on the node for which the alarm was generated.

      Log in to Manager, choose Hosts, and click the name of the host for which the alarm was generated. On the Dashboard tab page, at the Instances area, click a faulty instance. On the displayed instance page, choose More > Restart Instance. Repeat the preceding operations to restart all faulty instances on the host.

10. Wait 5 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 11.

Check whether disk space is sufficient.

11. On the FusionInsight Manager, check whether the alarm list contains ALM-12017 Insufficient Disk Capacity.
    • If yes, go to Step 12.
    • If no, go to Step 15.

12. Rectify the fault by following the steps provided in ALM-12017 Insufficient Disk Capacity.
13. Wait for 5 minutes. In the alarm list, check whether ALM-12017 Insufficient Disk Capacity is cleared.
    • If yes, go to Step 14.
    • If no, go to Step 15.

14. Wait for 5 minutes. In the alarm list, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 15.

Collect fault information.

15. On the FusionInsight Manager, choose O&M > Log > Download.
16. According to the service name obtained in Step 1, select the component and NodeAgent from the Service and click OK.
17. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
18. Contact the O&M personnel and send the collected log information.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None