Switching to ClickHouseKeeper

Impact on the System

ZooKeeper must be temporarily stopped during this operation.

Stopping components will make them unavailable, which may affect upper-layer services in the cluster. To minimize the impact, perform this operation during off-peak hours or after confirming that the operation does not have adverse impact.

Procedure

1. Log in to the ZooKeeper client node, go to the client installation directory, and configure environment variables.

   cd /opt/client

   source bigdata_env

2. If Kerberos authentication has been enabled for the current cluster, run the following command to authenticate the current user. For details, see Managing Roles to configure roles with required permissions. For details about how to bind roles with users, see Creating a User. If Kerberos authentication is disabled for the current cluster, skip this step:

   kinit MRS cluster user

   You must use a user with ZooKeeper service administrator permissions.

   Example: kinit zookeeperuser

3. Connect to ZooKeeper.

   zkCli.sh -server <IP address of the ZooKeeper installation node>:<port>

   Example: zkCli.sh -server node-master1DGhZ:2181

4. Set ACL permissions on the /clickhouse directory in ZooKeeper to allow read and write access for all users.

   setAcl -R /clickhouse world:anyone:cdrwa

5. Trigger ZooKeeper to generate the latest snapshot file:
   1. Check the IP address of the leader ZooKeeper instance.

      Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper, and record the name next to Host Where Leader Locates in the Basic Information area on the ZooKeeper Dashboard page. Click the Instances tab to check the node IP address of the host where the leader instance is located.

   2. Log in to the leader instance node, go to the /srv/BigData/zookeeper/version-2/ directory, and record the number of files in the directory.

      

   3. On the ZooKeeper Dashboard page, click Stop Service in the upper right corner. Stop the ZooKeeper service as prompted.
      

      Stopping components will make them unavailable, which may affect upper-layer services in the cluster. To minimize the impact, perform this operation during off-peak hours or after confirming that the operation does not have adverse impact.

   4. On the ZooKeeper service page, click the Instances tab, select the leader instance, and click Start Instance to start the instance.
   5. Select the leader instance and choose More > Stop Instance to stop the instance.
   6. Go to the ZooKeeper Dashboard page, and click Start Service in the upper right corner to start the ZooKeeper service.
   7. Log in to the leader instance node and check whether the latest snapshot file is generated.

      cd /srv/BigData/zookeeper/version-2/

      Compare the number of files recorded in 5.b and check whether the latest snapshot file has been generated, as shown in the following figure.

      

6. (Optional) If ClickHouseKeeper instances are not installed in the ClickHouse cluster, install them. For details, see section "Adding a Role Instance" in .
7. Log in to any ClickHouseServer node as user root and run the following commands to convert the ZooKeeper metadata into the format supported by ClickHouseKeeper:

   su - omm

   cd /opt/huawei/Bigdata/FusionInsight_ClickHouse_*/install/FusionInsight-ClickHouse-*-lts/clickhouse/sbin/

   sh clickhouse_transfer_zkmedata_to_keeper.sh

8. Restart the ClickHouseKeeper instances to load the migrated metadata.

   Log in to FusionInsight Manager and choose Cluster > Services > ClickHouse. Click the Instances tab. Select all ClickHouseKeeper instances, and choose More > Restart Instance.

9. Run the following command on a ClickHouseServer node to verify data consistency.

   sh clickhouse_transfer_zk2keeper_metadata_check.sh Username 'Password' Port

   • Use the service username and password of the ClickHouse service.
   • Use the HTTPS port (default: 21422) of ClickHouseServer for clusters in security mode, and the HTTP port (default: 21421) for clusters in normal mode. For details about how to obtain the port number, see ClickHouse Client Practices.

   If the verification is successful, the following information is displayed:

   ClickHouse meta check successfully.

   If the verification fails, check clickhouse_metadata_check_xxx.log in the current directory.

10. Update the ClickHouse configuration to use ClickHouseKeeper instead of ZooKeeper.

    sh clickhouse_transfer_zk2keeper_metadata_check.sh change_use_keeper_state true

11. Log in to FusionInsight Manager and choose Cluster > Services > ClickHouse. On the ClickHouse Dashboard page, choose More > Synchronize Configuration in the upper right corner. In the displayed dialog box, click OK to synchronize ClickHouse configurations.
12. Choose Cluster > Services > ClickHouse. On the displayed page, click the Instances tab, select all ClickHouseServer and ClickHouseBalancer instances, and choose More > Restart Instance to restart all ClickHouseServer and ClickHouseBalancer instances.
13. Verify that the ClickHouseKeeper has been used.

    On the ZooKeeper Dashboard page, click Stop Service in the upper right corner. Stop the ZooKeeper service as prompted. Once ZooKeeper is stopped, check the status of the ClickHouse service. If it remains operational, ClickHouse automatically transitions to ClickHouseKeeper.

Rollback Operations

If an error occurs during the switch to ClickHouseKeeper and cannot be resolved promptly, run the following command to revert the configuration to ZooKeeper. The ClickHouse service will continue using ZooKeeper for coordination.

1. Log in to any ClickHouseServer node as user root and run the following commands to perform the rollback:

   su - omm

   cd /opt/huawei/Bigdata/FusionInsight_ClickHouse_*/install/FusionInsight-ClickHouse-*-lts/clickhouse/sbin/

   sh clickhouse_transfer_zk2keeper_metadata_check.sh change_use_keeper_state false

2. Log in to FusionInsight Manager and choose Cluster > Services > ClickHouse. On the displayed page, choose More > Synchronize Configuration in the upper right corner. In the displayed dialog box, click OK to synchronize ClickHouse configurations.
3. Click the Instances tab, select all ClickHouseServer and ClickHouseBalancer instances, and choose More > Restart Instance to restart all ClickHouseServer and ClickHouseBalancer instances.
4. Select all ClickHouseKeeper instances and choose More > Stop Instance to stop the instances.
5. After confirming that the ClickHouse service is running normally, choose More > Delete Instance to delete ClickHouseKeeper instances.