ALM-45447 ClickHouse Table Read-Only

Alarm Description

The system checks the table status every minute. This alarm is generated when the system detects that a table is read-only. This alarm is automatically cleared when no table is read-only.

Alarm Attributes

Alarm Parameters

Impact on the System

• Data cannot be written to or modified.
• Data synchronization in the replication table is interrupted, causing data inconsistency.

Possible Causes

The ZooKeeper is overloaded and metadata is lost.

Handling Procedure

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address of the hostname in Location.
2. Log in to the node where the client is installed and run the following commands:

   cd {Client installation path}

   source bigdata_env

   • Security mode (with Kerberos enabled):

     kinit Component service user

     clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 21427 --secure

   • Normal mode (with Kerberos disabled):

     clickhouse client --host IP address of the ClickHouseServer instance for which the alarm is reported --user Username --password --port 21423

3. Run the following SQL statement to check whether any table is in the read-only state:

   select database,table from system.replicas where is_readonly = 1

   • If yes, go to Step 4.
   • If no, go to Step 8.

4. Specify the database and table queried in Step 3 in the following SQL statements and run them in sequence. Then, run the SQL statement in Step 3 and check whether the result contains the read-only table.

   system restore replica database.table;

   system restart replica database.table;

   • If yes, go to Step 5.
   • If no, go to Step 8.

5. Specify the database and table queried in Step 3 in the following SQL statements and run them in sequence. Then, run the SQL statement in Step 3 and check whether the result contains the read-only table.

   detach table database.table;

   attach table database.table;

   • If yes, go to Step 6.
   • If no, go to Step 8.

6. Run the following SQL statement to view the structure information of the read-only table. In the statement, database and table are those queried in Step 3.

   show create table database.table;

7. Run the following SQL statement to delete the read-only table and create a read-only table based on the table structure information in Step 6. Wait for several minutes, run the SQL statement in Step 3, and check whether the result contains the read-only table.

   drop table database.table no delay;

   • If yes, go to Step 9.
   • If no, go to Step 8.

   

8. Wait several minutes and check whether the alarm is automatically cleared.
   • If yes, no further action is required.
   • If no, go to Step 9.

Collect fault information.

9. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
10. Expand the Service drop-down list, and select ClickHouse for the target cluster.
11. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
12. Click the edit icon in the upper right corner and select a time span starting 1 hour before and ending 1 hour after when the alarm was generated. Then, click Download to collect the logs.
13. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None