How Do I Quickly Restore the Status of a Logical Cluster in a Scale-in Fault Scenario?

Symptom

After the scale-in of the ClickHouse logical cluster is complete, Status is displayed as Scaling in for a long time.

Procedure

1. Log in to FusionInsight Manager, choose Cluster > Services > ClickHouse, and click Logical Cluster. After the cluster scale-in is complete, check whether Status is Scaling in for a long time.
   • If yes, go to 2.
   • If no, the scale-in of the logical cluster is complete. No further action is required.

2. Log in to the node where the ZooKeeper client is installed in the MRS cluster and run the following commands:

   su - omm

   source {Client installation directory}/bigdata_env

   kinit Component user (You do not need to run the kinit command for normal clusters.)

   zkCli.sh -server {Service IP address of the ZooKeeper service instance node}:{Client port number}

   get /clickhouse/logic_cluster

   Check whether the value of <status> is REDUCING.

   • If yes, go to 3.
   • If no, contact technical support to further analyze the customized UI log information to locate the fault.
   

   The ClickHouse metadata root directory in ZooKeeper varies depending on ClickHouse multi-service scenarios.

   Log in to FusionInsight Manager and choose Cluster > Services > ClickHouse. Click Configurations then All Configurations, and query the value of clickhouse.zookeeper.root.path.

3. Run the following command to update the <status> value of the logical cluster to CHECKING:

   set /clickhouse/logic_cluster <clusters><default_cluster><createTime>……</createTime><sslBalancerPort>……</sslBalancerPort><balancerPort>……</balancerPort><httpsBalancerPort>……</httpsBalancerPort><httpBalancerPort>……</httpBalancerPort><replicaNum>……</replicaNum><status>CHECKING</status><node>……</node><node>……</node></default_cluster></clusters>

   

   To modify the ZNode information in ZooKeeper, convert all content into one line and change the value of <status> to CHECKING.