ALM-12061 Process Usage Exceeds the Threshold

Description

The system checks the usage of the omm process every 30 seconds. Users can run the ps -o nlwp, pid, args, -u omm | awk '{sum+=$1} END {print "", sum}' command to obtain the number of concurrent processes of user omm. Run the ulimit -ucommand to obtain the maximum number of processes that can be simultaneously opened by user omm. Divide the number of concurrent processes by the maximum number to obtain the process usage of user omm. The process usage has a default threshold. This alarm is generated when the process usage exceeds the threshold.

If Trigger Count is 3 and the process usage is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1and the process usage is less than or equal to 90% of the threshold, this alarm is cleared.

Attribute

Parameters

Impact on the System

Service failure: When the process usage exceeds the threshold, you cannot switch to user omm. A new omm thread cannot be created. As a result, the job may fail to run.

Possible Causes

• The alarm threshold is improperly configured.
• The maximum number of processes (including threads) that can be concurrently opened by user omm is inappropriate.
• An excessive number of threads are opened at the same time.

Procedure

Check whether the alarm threshold or alarm hit number is properly configured.

1. On the FusionInsight Manager, change the alarm threshold and Trigger Count based on the actual CPU usage.

   Specifically, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host> Process > omm Process Usage to change Trigger Count, as shown in Figure 1.

   

   The alarm is generated when the process usage exceeds the threshold for the times specified by Trigger Count.

   Figure 1 Setting Trigger Count
   

   Set the alarm threshold based on the actual process usage. To check the process usage, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host> Process > omm Process Usage, as shown in Figure 2.

   Figure 2 Setting an alarm threshold
   

2. 2 minutes later, check whether the alarm is cleared.
   • If it is, no further action is required.
   • If it is not, go to 3.

Check whether the maximum number of processes (including threads) opened by user omm is appropriate.

3. In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and view the IP address of the host for which the alarm is generated.
4. Log in to the host where the alarm is generated as user root.
5. Run the su - omm command to switch to user omm.
6. Run the ulimit -u command to obtain the maximum number of threads that can be concurrently opened by user omm and check whether the number is greater than or equal to 60000.
   • If it is, go to 8.
   • If it is not, go to 7.

7. Run the ulimit -u 60000 command to change the maximum number to 60000. Two minutes later, check whether the alarm is cleared.
   • If it is, no further action is required.
   • If it is not, go to 12.

Check whether an excessive number of processes are opened at the same time.

8. In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and view the IP address of the host for which the alarm is generated.
9. Log in to the host where the alarm is generated as user root.
10. Run the ps -o nlwp, pid, lwp, args, -u omm|sort -n command to check the numbers of threads used by the system.

    The result is sorted based on the thread number. Analyze the top 5 thread numbers and check whether the threads are incorrectly used.

    • If it is, perform the following operations to stop the parent processes of the top 5 abnormally used processes without affecting services, go to 11.
      1. Run the following command to query the parent process of the corresponding process:

         ps -ef | grep "Process ID"

         The third column in the command output is the parent process ID.

      2. Run the following command to stop the parent process:

         kill -9 parent process ID

    • If it is not, run the ulimit -u command to change the maximum number to be greater than 60000.

11. Five minutes later, check whether the alarm is cleared.
    • If it is, no further action is required.
    • If it is not, go to 12.

Collect fault information.

12. On the FusionInsight Manager home page of the active clusters, choose O&M > Log > Download.
13. Select OmmServer and NodeAgent from the Service and click OK.
14. Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
15. Contact the O&M personnel and send the collected log information.

Alarm Clearing

This alarm will be automatically cleared after the fault is rectified.

Related Information

None