ALM-45425 ClickHouse Service Unavailable

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The alarm module checks the ClickHouseServer instance status every 60 seconds. This alarm is generated when the alarm module detects that all instances of a ClickHouseServer shard are abnormal.

This alarm is cleared when the system detects that any instance of the ClickHouseServer shard is restored and the alarm is handled.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
45425	Critical	Quality of service	ClickHouse	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster or system for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.

Impact on the System

FusionInsight Manager cannot be used to perform cluster operations on the ClickHouse service, and ClickHouse service functions are unavailable.

Possible Causes

The ZooKeeper service is unavailable.
The configuration information in the metrika.xml file in the component configuration directory of the faulty ClickHouse instance is inconsistent with configuration of that ClickHouse instance in ZooKeeper.

Handling Procedure

The ZooKeeper service is unavailable.

Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. In the alarm list, check whether alarm ALM-13000 ZooKeeper Service Unavailable exists.
- If yes, go to 2.
- If no, go to 3.
Handle alarm ALM-13000 ZooKeeper Service Unavailable according to the alarm help. In the alarm list, check whether alarm ALM-45425 ClickHouse Service Unavailable is cleared.
- If yes, no further action is required.
- If no, go to 3.

Check whether the configuration in metrika.xml of the ClickHouse instance is correct.

Log in to FusionInsight Manager, choose Cluster > Services > ClickHouse, click the Instance tab, and check whether there is any abnormal ClickHouse instance based on the alarm information.
- If yes, go to 4.
- If no, go to 11.
Log in to the host where the ClickHouse service is abnormal and ping the IP address of another normal ClickHouse instance node to check whether the network connection is normal.
- If yes, go to 5.
- If no, contact the network administrator to repair the network.

Log in to the node where the client is installed as the client installation user and run the following commands:

cd {Client installation path}

source bigdata_env
- For a cluster with Kerberos authentication enabled (security mode):
  kinit Component service user
  
  clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure
- For a cluster with Kerberos authentication disabled (normal mode):
  clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000
Run the following command to query the value of macros.id:

select substitution from system.macros where macro='id';
Log in to the host where the ZooKeeper client is located and log in to the ZooKeeper client.

Switch to the client installation directory.

Example: cd /opt/client

Run the following command to configure environment variables:

source bigdata_env

Run the following command to authenticate the user (skip this step for a cluster in normal mode):

kinit Component service user

Run the following command to log in to the client tool:

zkCli.sh -server Service IP address of the node where the ZooKeeper instance resides: Client port
Run the following command to check whether the ClickHouse cluster topology information can be obtained:

get /clickhouse/config/macros.id value in 5/metrika.xml
- If yes, go to 8.
- If no, go to 11.

Log in to the host where the ClickHouse instance is abnormal and go to the configuration directory of the ClickHouse instance.

cd ${BIGDATA_HOME}/FusionInsight_ClickHouse_Version/x_x_ClickHouseServer/etc

cat metrika.xml
Check whether the cluster topology information on ZooKeeper obtained in 7 is the same as that in the metrika.xml file in the component configuration directory in 8.
- If yes, check whether the alarm is cleared. If the alarm persists, go to 11.
- If no, go to 10.
On FusionInsight Manager, choose Cluster > Services > ClickHouse, click More, and select Synchronize Configuration. Then, check whether the service status is normal and whether the alarm is cleared 5 minutes later.
- If yes, no further action is required.
- If no, go to 11.

Collect fault information.

On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the Service drop-down list, and select ClickHouse for the target cluster.
Expand the Hosts drop-down list and select the target hosts.
Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.

Parent topic: Alarm Reference

Previous topic: ALM-45293 Ranger User Synchronization Exception

Next topic: ALM-45426 ClickHouse Service Quantity Quota Usage in ZooKeeper Exceeds the Threshold

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot