How Do I Quickly Restore ClickHouse When Concurrent Requests Are Stacked for a Long Time?

Symptom

When there are too many concurrent ClickHouse requests, new requests cannot be handled, affecting service availability. The error information is as follows:

DB::Exception: Too many simultaneous queries. Maximum: 100

Procedure

1. Log in to the ClickHouse client node and connect to the server where the error is reported. For details, see ClickHouse Client Practices.
2. Run the following command to check the number of concurrent SQL jobs that are being executed:

   select count(*) from system.processes;

   Verify that the obtained value is greater than or equal to 100.

3. Run the following command to stop large query SQL statements that are being executed:

   kill query where query_kind='Select' and elapsed > 60;

4. Wait for 30 seconds and run the following command to collect statistics on the SQL jobs that are being executed:

   select count(*) from system.processes;

   Check whether the obtained value is less than 60.

5. If the number of concurrent jobs is greater than 60, log in to FusionInsight Manager, choose Cluster > Services > ClickHouse > Instances, select the ClickHouseServer instance that reports the error, and choose More > Restart Instance.