On this page

Show all

Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Alarm Reference/ ALM-45426 ClickHouse Service Quantity Quota Usage in ZooKeeper Exceeds the Threshold

ALM-45426 ClickHouse Service Quantity Quota Usage in ZooKeeper Exceeds the Threshold

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The alarm module checks the quota usage of the ClickHouse service in the ZooKeeper every 60 seconds. This alarm is generated when the alarm module detects that the usage exceeds the threshold (90%).

This alarm is cleared when the system detects that the usage is lower than the threshold and the alarm is cleared.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
45426	Critical (default threshold: 95%) Major (default threshold: 90%)	Quality of service	ClickHouse	Yes

Alarm ID

Alarm Severity

Alarm Type

Service Type

Auto Cleared

45426

Critical (default threshold: 95%)

Major (default threshold: 90%)

Quality of service

ClickHouse

Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster or system for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.

Impact on the System

After the ZooKeeper quantity quota of the ClickHouse service exceeds the threshold, you cannot perform cluster operations on the ClickHouse service on FusionInsight Manager. As a result, the ClickHouse service cannot be used.

Possible Causes

When table data is created, inserted, or deleted, the ClickHouse creates znodes on ZooKeeper nodes. As the service volume increases, the number of znodes may exceed the configured threshold.
No quota limit is set for the metadata directory /clickhouse of ClickHouse in ZooKeeper.

Handling Procedure

Check the number of znodes created by ClickHouse on ZooKeeper.

Log in to the host where the ZooKeeper client is located and log in to the ZooKeeper client.

Switch to the client installation directory.

Example: cd /opt/client

Run the following command to configure environment variables:

source bigdata_env

Run the following command to authenticate the user (skip this step in common mode):

kinit Component service user

Run the following command to log in to the client tool:

zkCli.sh -server service IP address of the node where the ZooKeeper role instance locates:client port
Run the following command to check the quota used by the ClickHouse in the ZooKeeper and check whether the quota information is correctly set:
listquota /clickhouse
```
absolute path is /zookeeper/quota/clickhouse
Quota for path /clickhouse does not exist.
```
If the preceding information indicates that the quota configuration is incorrect, go to 3.

If no, go to 5.
Log in to FusionInsight Manager and choose Cluster > Services > ZooKeeper. On the displayed page, click Configurations and click All Configurations. On this sub-tab page, search for quotas.auto.check.enable to check whether its value is true.

If the value is not true, change the value to true and click Save.
On FusionInsight Manager, choose Cluster > Services > ClickHouse, click More, and select Synchronize Configuration. After the synchronization is successful, go to 1.
Run the following command and check whether the ratio of the count value of Output stat to the count value of Output quota in the command output is greater than 0.9:
listquota /clickhouse
```
absolute path is /zookeeper/quota/clickhouse
Output quota for /clickhouse count=200000,bytes=1000000000
Output stat for /clickhouse count=2667,bytes=60063
```
In the preceding information, the count value of Output stat is 2667, and the count value of Output quota is 200000.
- If yes, go to 6.
- If no, check whether the alarm is cleared 5 minutes later. If the alarm persists, go to 8.
On FusionInsight Manager, choose Cluster > Services > ClickHouse. Click Configurations and then All Configurations. Search for clickhouse.zookeeper.quota.node.count, set it to a value twice the count of Output stat in 5. Do not use a value larger than 6000000. Otherwise, there will be high risks. Exercise caution when setting this parameter.
Restart the ClickHouse instance for which the alarm is generated, and check whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, perform 6 again, and check whether the alarm is cleared 5 minutes later. If the alarm persists, go to 8.

Collect the fault information.

On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the Service drop-down list, and select ClickHouse for the target cluster.
Choose the corresponding host form the host list.
Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.

Parent topic: Alarm Reference

Previous topic: ALM-45425 ClickHouse Service Unavailable

Next topic: ALM-45427 ClickHouse Service Capacity Quota Usage in ZooKeeper Exceeds the Threshold

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot