Error Message "connection established slowly"

Scenario

During peak hours, the connection between a client and an RDS for MySQL instance often times out. As a result, it takes more than 10 seconds to log in to the instance.

Possible Causes

1. Check error logs of the instance to check whether the information "connection xxx is established slowly" is displayed. Example:

   

   If yes, some connections have timed out and have not been processed by the RDS for MySQL instance yet. When the connection between a client and the instance exceeds the specified timeout duration, an error is reported.

2. Check the thread pool configuration (enabled by default) on the console.

   In the preceding figure, threadpool_size is set to 1, threadpool_stall_limit is set to 500 ms, and threadpool_oversubscribe is set to 3. The wait time for the thread pool to process connections is mainly related to the three parameters.

   • When all threads in the thread pool are busy, the scheduling thread in the thread pool creates a new thread every 500 ms (threadpool_stall_limit). In this case, each thread group can process a new connection every 500 ms on average. If the queue is too long, the client may time out.
   • If all threads are busy, the number of working threads reaches the total number of threads in the thread pool. When a large number of connections are established, the total number of threads is calculated as threadpool_size x (threadpool_oversubscribe + 1).
3. Therefore, if the value of threadpool_size is improper or too small, new connections will be established slowly during peak hours.

Solution

If there are a large number of new connections, increase the values of threadpool_size and threadpool_oversubscribe to increase the total number of threads and decrease the value of threadpool_stall_limit to reduce the wait time for new connections. This reduces the overhead of repeated thread creation and destruction, and limits the number of running threads to protect the system against avalanche.

In normal cases, the thread pool is used when there are a large number of short connections. If persistent connections are used and there are a few connections (for example, the client uses a connection pool), the thread pool is not so helpful. In this case, adjust the values of threadpool_size and threadpool_oversubscribe to increase the total number of threads, or close the thread pool.