Help Center > > User Guide> FusionInsight Manager Operation Guide (Applicable to 3.x)> Alarm Reference (Applicable to MRS 3.x)> ALM-45427 ClickHouse Service Capacity Quota Usage in ZooKeeper Exceeds the Threshold

ALM-45427 ClickHouse Service Capacity Quota Usage in ZooKeeper Exceeds the Threshold

Updated at: Sep 02, 2021 GMT+08:00

Description

The alarm module checks the quota usage of the ClickHouse service in the ZooKeeper every 60 seconds. This alarm is generated when the alarm module detects that the usage exceeds the threshold (90%).

This alarm is cleared when the system detects that the usage is lower than the threshold and the alarm is cleared.

Attribute

Alarm ID

Alarm Severity

Auto Clear

45427

Major (default)

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the name of the service for which the alarm is generated.

RoleName

Specifies the name of the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

After the ZooKeeper quantity quota of the ClickHouse service exceeds the threshold, you cannot perform cluster operations on the ClickHouse service on FusionInsight Manager. As a result, the ClickHouse service cannot be used.

Possible Causes

When table data is created, inserted, or deleted, the ClickHouse creates znodes on ZooKeeper nodes. As the service volume increases, the capacity of znodes may exceed the configured threshold.

Procedure

Check the znode capacity of the ClickHouse in the ZooKeeper.

  1. Log in to the host where the ZooKeeper client is located and log in to the ZooKeeper client.

    Switch to the client installation directory.

    Example: cd /opt/client

    Run the following command to configure environment variables:

    source bigdata_env

    Run the following command to authenticate the user (skip this step in common mode):

    kinit Component service user

    Run the following command to log in to the client tool:

    zkCli.sh -server service IP address of the node where the ZooKeeper role instance locates:client port

  2. Run the following command to check the quota used by ClickHouse on ZooKeeper and check whether the ratio of the bytes value of Output stat to the bytes value of Output quota in the command output is greater than 0.9.

    listquota /clickhouse
    absolute path is /zookeeper/quota/clickhouse
    Output quota for /clickhouse count=200000,bytes=1000000000
    Output stat for /clickhouse count=2667,bytes=60063
    In the preceding information, the bytes value of Output stat is 60063, and the bytes value of Output quota is 1000000000.
    • If yes, go to 4.
    • If no, wait for 5 minutes and check whether the alarm is cleared. If the alarm persists, go to 5.

  3. On FusionInsight Manager, choose Cluster > Services > ClickHouse > Configurations > All Configurations, search for the clickhouse.zookeeper.quota.size parameter, and change the value of this parameter to twice the bytes value of Output stat in 2.
  4. Restart the ClickHouse instance for which the alarm is generated, wait for 5 minutes, and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, perform 4 again, wait for 5 minutes, and check whether the alarm is cleared. If the alarm persists, go to 5.

Collect the fault information.

  1. On FusionInsight Manager, choose O&M > Log > Download.
  2. Expand the Service drop-down list, and select ClickHouse for the target cluster.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact O&M personnel and send the collected logs.

Alarm Clearing

After the fault that triggers the alarm is rectified, the alarm is automatically cleared.

Related Information

None

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel