ALM-45445 Failed to Send Data Files to Remote Shards When ClickHouse Writes Data to a Distributed Table

Alarm Description

The ClickHouse instance checks the distributed table every 300 seconds. If the number of consecutive failures exceeds the threshold, an alarm is generated. In this case, the node where the ClickHouse instance writes data to the distributed table cannot send data files to remote shard nodes.

This alarm is cleared when the number of consecutive failures falls below the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

The results of operations such as distributed table queries are abnormal.

Possible Causes

The status of some ClickHouse shard nodes is abnormal.

Handling Procedure

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address of the hostname in Location.
2. Log in to the node where the client is installed as the client installation user and run the following commands:

   cd {Client installation path}

   source bigdata_env

   • For a cluster with Kerberos authentication enabled (security mode):

     kinit Component service user

     clickhouse client --host IP address of the ClickHouseServer instance for which the alarm is reported --port 9440 --secure

   • For a cluster with Kerberos authentication disabled (normal mode):

     clickhouse client --host IP address of the ClickHouseServer instance for which the alarm is reported --user Username --password --port 9000

3. Run the following SQL statement to obtain the shard based on the value of data_path. For example, if the value of data_path is /srv/Bigdata/clickhouse/data1.../shard2_all_replicas, the desired shard is shard2.

   select database, table, data_path, data_files, error_count from system.distribution_queue where data_files != 0 and error_count != 0;

4. Run the following SQL statement to obtain the node IP address (value of the host field in the system table system.clusters) of the shard where data fails to be sent to (shard_num obtained in 3):

   select * from system.clusters;

5. Log in to the ClickHouse node (IP address obtained in 2) by referring to 4, run the following statement, and check whether the result can be returned:

   SELECT 1;

   If yes, go to 6.

   If no, go to 10.

6. Log in to the ClickHouse node (IP address obtained in 2) by referring to 4, run the following statement (database_name and table_name indicate the database name and table name of the local table corresponding to the distributed table):

   select name,type from system.columns where database='database_name' and table='table_name'

7. Log in to the ClickHouse node for which the alarm is generated by referring to 2 and run the following statement (database_name and table_name indicate the database name and table name of the distributed table to which data is written):

   select name,type from system.columns where database='database_name' and table='table_name'

8. Check whether the results obtained in 6 and 7 are the same.

   If yes, go to 10.

   If no, ensure that the column information of the distributed table is consistent with that of the corresponding local table, and then try to write data to the distributed table.

9. Wait for several minutes and check whether the alarm is cleared.

   If yes, no further action is required.

   If no, go to 10.

Collect fault information.

10. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
11. Expand the Service drop-down list, and select ClickHouse for the target cluster.
12. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
13. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
14. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.