Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-45444 Abnormal ClickHouse Process
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-45444 Abnormal ClickHouse Process

Alarm Description

The health check module checks ClickHouse instances every 30 seconds. If the number of consecutive failures exceeds the threshold, an alarm is reported. In this case, the ClickHouse process may stop responding and services cannot be properly executed.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

45444

Critical

Yes

Alarm Parameters

Parameter

Description

Source

Specifies the cluster or system for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

If the ClickHouse process is abnormal, services cannot run properly.

Possible Causes

The ClickHouse process runs improperly.

Handling Procedure

  1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
  2. Log in to the node where the client is installed as the client installation user and run the following commands:

    cd {Client installation path}

    source bigdata_env

    • For a cluster with Kerberos authentication enabled (security mode):

      kinit Component service user

      clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure

    • For a cluster with Kerberos authentication disabled (normal mode):

      clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user Username --password --port 9000

  3. Run the following statement to check whether the result can be properly returned:

    SELECT 1;

    • If yes, go to 4.
    • If no, go to 5.

  4. Wait for several minutes and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 5.

Collect fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the Service drop-down list, and select ClickHouse for the target cluster.
  3. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
  4. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
  5. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.