ALM-45447 ClickHouse Table Read-Only

Alarm Description

The system checks the table status every minute. This alarm is generated when the system detects that a table is read-only. This alarm is automatically cleared when no table is read-only.

Alarm Attributes

Alarm Parameters

Impact on the System

• Data cannot be written to or modified.
• Data synchronization in the replication table is interrupted, causing data inconsistency.

Possible Causes

The ZooKeeper is overloaded and metadata is lost.

Handling Procedure

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address of the hostname in Location.
2. Log in to the node where the client is installed and run the following commands:

   cd {Client installation path}

   source bigdata_env

   • Security mode (with Kerberos enabled):

     kinit Component service user

     clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 21427 --secure

   • Normal mode (with Kerberos disabled):

     clickhouse client --host IP address of the ClickHouseServer instance for which the alarm is reported --user Username --password --port 21423

3. Run the following SQL statement to check whether any table is in the read-only state:

   select database,table from system.replicas where is_readonly = 1

   • If yes, go to 4.
   • If no, go to 8.

4. Specify the database and table queried in 3 in the following statements and run them in sequence. Then, run the SQL statement in 3 and check whether the result contains any read-only table.

   system restore replica database.table;

   system restart replica database.table;

   • If yes, go to 5.
   • If no, go to 8.

5. Specify the database and table queried in 3 in the following statements and run them in sequence. Then, run the SQL statement in 3 and check whether the result contains the read-only table.

   detach table database.table;

   attach table database.table;

   • If yes, go to 6.
   • If no, go to 8.

6. Run the following SQL statement to view the structure information of the read-only table. In the statement, database and table are those queried in 3.

   show create table database.table;

7. Run the following SQL statement to delete the read-only table and create a read-only table based on the table structure information in 6. Wait for several minutes, run the SQL statement in 3, and check whether the result contains the read-only table.

   drop database.table no delay;

   • If yes, go to 9.
   • If no, go to 8.

   

8. Wait several minutes and check whether the alarm is automatically cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Collect fault information.

9. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
10. Expand the Service drop-down list, and select ClickHouse for the target cluster.
11. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
12. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
13. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.