ALM-45459 ClickHouse Failed to Set the Table to Read-Only

Alarm Description

This alarm is generated when the read-only function of the standby cluster is enabled in the DR cluster but the ClickHouse DR table fails to be set to read-only.

This alarm is cleared when the DR table is successfully set to read-only.

This section applies only to MRS 3.6.0-LTS.1 or later.

Alarm Attributes

Alarm Parameters

Impact on the System

If the table of the standby ClickHouse cluster fails to be set to read-only, writing data to the DR table in the DR cluster will cause DR data damage.

Possible Causes

• The target table does not exist on the ClickHouseServer node.
• The ClickHouseServer has reached its maximum concurrency limit.

Handling Procedure

Check whether the target table exists on the ClickHouseServer node.

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address of the hostname in Location.
2. Log in to the node queried in Step 1 as the root user and run the following commands to check the table that fails to be set to read-only:

   su - omm

   cd ${BIGDATA_TMP}/disaster/ClickHouse/readOnly

   more failedTables.txt

   

   In the preceding command, ClickHouse indicates the internal service name. In the multi-service scenario, it may be ClickHouse-1,ClickHouse-2,...

3. Log in to the node where the client is installed as the client installation user and run the following commands:

   cd {Client installation path}

   source bigdata_env

   • For a cluster with Kerberos authentication enabled (security mode):

     kinit Component service user

     clickhouse client --host IP address of the ClickHouseServer instance for which the alarm was reported --port 9440 --secure

   • For a cluster with Kerberos authentication disabled (normal mode):

     clickhouse client --host IP address of the ClickHouseServer instance for which the alarm was reported --user Username --password --port 9440

4. Run the following statement to check whether the table in Step 2 exists:

   select database||'.'||name from system.tables where database = 'Database name' and name = 'Table name';

   • If yes, go to Step 5.
   • If no, go to Step 7.

5. Log in to the node queried in Step 1 as the root user and run the following command to check whether the maximum concurrency limit is reached:

   grep -i "too many" /var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.log | tail -n 3;

   • If yes, go to Step 6.
   • If no, go to Step 7.

6. Log in to FusionInsight Manager of the active cluster, choose Active/Standby Cluster DR > Details > Notifying the DR status, enter the user password, and click OK. Wait for three minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
8. Expand the Service drop-down list and select ClickHouse for the target cluster.
9. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
10. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
11. Send the collected fault logs to O&M engineers for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None