Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-16051 Percentage of Sessions Connected to MetaStore Exceeds the Threshold
Updated on 2024-11-13 GMT+08:00
View PDF
Share

ALM-16051 Percentage of Sessions Connected to MetaStore Exceeds the Threshold

Alarm Description

The system checks the percentage of sessions connected to MetaStore to the maximum number of sessions allowed by MetaStore every 30 seconds. This alarm is generated when the percentage exceeds the threshold.

This alarm is cleared when the percentage of MetaStore sessions is less than or equal to the threshold.

This alarm applies to MRS 3.3.1 or later.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

16051

Critical (default threshold: 90%)

Major (default threshold: 80%)

Yes

Alarm Parameters

Type

Parameter

Description

Location Information

Source

Specifies the cluster for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

HostName

Specifies the host for which the alarm was generated.

Additional Information

Trigger condition

Specifies the alarm triggering condition.

Impact on the System

If this alarm is generated, sessions connected to MetaStore are too many. As a result, new connections cannot be set up.

Possible Causes

Too many clients are connected to MetaStore.

Handling Procedure

Change the maximum number of MetaStore connections.

  1. On FusionInsight Manager, choose Cluster > Services > Hive, click Configuration and then All Configurations.
  2. In the All Configurations tab, search for hive.metastore.server.max.threads and check whether the value is the maximum 10000.

    • If yes, go to 6.
    • If no, go to 3.

  3. Change the value of hive.metastore.server.max.threads to 10000 and click Save.
  4. Click Instances, select all MetaStore instances, and choose More > Restart Instance.

    During MetaStore instance restart, the instance cannot provide services for external systems. SQL tasks that are being executed on the instance may fail.

  5. Check whether this alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

Collect fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the Service drop-down list, and select Hive for the target cluster.
  3. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. On FusionInsight Manager, choose Cluster > Services > Hive. On the displayed Dashboard page, click More and select Collect Stack Information. On the displayed page, set the following parameters:

    • Select MetaStore for the role where you want to collect data.
    • Select jstack and Enable continuous collection of jstack and jmap -histo information.
    • Set the collection interval to 10 seconds and the duration to 2 minutes.

  5. Click OK. After the collection is complete, click Download.
  6. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.