On this page

Show all

Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Alarm Reference (Applicable to MRS 3.x)/ ALM-12061 Process Usage Exceeds the Threshold

ALM-12061 Process Usage Exceeds the Threshold

Updated on 2022-12-14 GMT+08:00

View PDF

Description

The system checks the usage of the omm process every 30 seconds. Users can run the ps -o nlwp, pid, args, -u omm | awk '{sum+=$1} END {print "", sum}' command to obtain the number of concurrent processes of user omm. Run the ulimit -ucommand to obtain the maximum number of processes that can be simultaneously opened by user omm. Divide the number of concurrent processes by the maximum number to obtain the process usage of user omm. The process usage has a default threshold. This alarm is generated when the process usage exceeds the threshold.

If Trigger Count is 3 and the process usage is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1and the process usage is less than or equal to 90% of the threshold, this alarm is cleared.

Attribute

Alarm ID	Alarm Severity	Auto Clear
12061	Major	Yes

Parameters

Name	Meaning
Source	Specifies the cluster or system for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
Trigger Condition	Specifies the threshold for triggering the alarm.

Impact on the System

Switch to user omm fails.
New omm process cannot be created.

Possible Causes

The alarm threshold is improperly configured.
The maximum number of processes (including threads) that can be concurrently opened by user omm is inappropriate.
An excessive number of threads are opened at the same time.

Procedure

Check whether the alarm threshold or alarm hit number is properly configured.

On the FusionInsight Manager, change the alarm threshold and Trigger Count based on the actual CPU usage.

Specifically, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host> Process > omm Process Usage to change Trigger Count, as shown in Figure 1.

NOTE:

The alarm is generated when the process usage exceeds the threshold for the times specified by Trigger Count.

Figure 1 Setting Trigger Count

Set the alarm threshold based on the actual process usage. To check the process usage, choose O&M > Alarm > Thresholds > Name of the desired cluster > Host> Process > omm Process Usage, as shown in Figure 2.

Figure 2 Setting an alarm threshold
2 minutes later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 3.

Check whether the maximum number of processes (including threads) opened by user omm is appropriate.

In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and view the IP address of the host for which the alarm is generated.
Log in to the host where the alarm is generated as user root.
Run the su - omm command to switch to user omm.
Run the ulimit -u command to obtain the maximum number of threads that can be concurrently opened by user omm and check whether the number is greater than or equal to 60000.
- If it is, go to 8.
- If it is not, go to 7.
Run the ulimit -u 60000 command to change the maximum number to 60000. Two minutes later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 12.

Check whether an excessive number of processes are opened at the same time.

In the alarm list on FusionInsight Manager, locate the row that contains the alarm, and view the IP address of the host for which the alarm is generated.
Log in to the host where the alarm is generated as user root.
Run the ps -o nlwp, pid, lwp, args, -u omm|sort -n command to check the numbers of threads used by the system. The result is sorted based on the thread number. Analyze the top 5 thread numbers and check whether the threads are incorrectly used. If they are, contact maintenance personnel to rectify the fault. If they are not, run the ulimit -u command to change the maximum number to be greater than 60000.
Five minutes later, check whether the alarm is cleared.
- If it is, no further action is required.
- If it is not, go to 12.

Collect fault information.

On the FusionInsight Manager home page of the active clusters, choose O&M > Log > Download.
Select OmmServer and NodeAgent from the Service and click OK.
Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
Contact the O&M personnel and send the collected log information.

Alarm Clearing

This alarm will be automatically cleared after the fault is rectified.

Related Information

None

Parent topic: Alarm Reference (Applicable to MRS 3.x)

Previous topic: ALM-12057 Metadata Not Configured with the Task to Periodically Back Up Data to a Third-Party Server

Next topic: ALM-12062 OMS Parameter Configurations Mismatch with the Cluster Scale

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot