Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-45448 Rapid Increase of Znodes Used by ClickHouse
Updated on 2024-11-13 GMT+08:00
View PDF
Share

ALM-45448 Rapid Increase of Znodes Used by ClickHouse

This section is available for MRS 3.3.1 or later version only.

Alarm Description

Metadata in ClickHouse is stored on ZooKeeper Znodes. The number of occupied Znodes may increase sharply even if service requirements do not change greatly. This alarm is generated when the number of occupied Znodes in two hours exceeds the threshold. If a large amount of data is imported or services are migrated, ignore this alarm.

This alarm is cleared when the system detects that the increase in two hours is lower than the threshold.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

45448

Major

Yes

Alarm Parameters

Type

Parameter

Description

Location Information

Source

Specifies the cluster or system for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

HostName

Specifies the host for which the alarm was generated.

Impact on the System

ZooKeeper server resources are occupied. The number of Znodes will reach the upper limit in a short period of time, affecting the ClickHouse service.

Possible Causes

  • If the ClickHouse service volume retains at large scale, increase the value of ZNODE_GROWTH_LIMIT on the ClickHouse configuration page.
  • Major service changes, migration, and data import have been performed in ClickHouse.

Handling Procedure

  1. Choose Cluster > ClickHouse > Instances > All Configurations, and search for ZNODE_GROWTH_LIMIT, increase the value of ZNODE_GROWTH_LIMIT. The default value is 100000, the maximum value is 200000, and the value is configured based a step of 50000. Wait for 2 hours and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 2.

  2. Confirm with the service party whether there are new service requests or a large amount of data is imported or migrated during the period when the alarm is reported.

    • If yes, go to 3.
    • If no, go to 4.

  3. Wait for 2 hours and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 4.

Collect fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the Service drop-down list, and select ClickHouse for the target cluster.
  3. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
  4. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
  5. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.