"Connection reset by peer" Is Returned When Using Elasticsearch with Spring Boot

Symptom

When a Spring Boot application uses the Java High Level REST Client to connect to an Elasticsearch cluster, the error Connection reset by peer is reported, the TCP connection is interrupted, and write failures occur.

Possible Causes

There are many possible causes. For example, the network went down; a firewall, switch, or VPN in the network was faulty; the keepalive settings were incorrect; the connected server node was changed; or the network was unstable.

Procedure

Use one of the following methods to solve the issue:

• Method 1

  Modify the timeout interval of RestHighLevelClient connection requests. The default value is 1000 ms. You can increase the value to 10,000 ms.

  RestClientBuilder builder = RestClient.builder(new HttpHost(endpoint, port))
          .setHttpClientConfigCallback(httpClientBuilder->    
  httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider))
          .setRequestConfigCallback(requestConfigBuilder ->
  requestConfigBuilder.setConnectTimeout(10000).setSocketTimeout(60000));
      return new RestHighLevelClient(builder);

  Settings for a single request: request.timeout(TimeValue.timeValueSeconds(60)).

• Method 2

  Set the RestHighLevelClient keepalive time to 15 minutes.

• Method 3

  Handle the exception captured in the code and retry the request.

Related Information

• TCP connections

  TCP connections are classified into persistent connections and short connections. A short TCP connection is automatically disconnected after data packets are sent. A persistent TCP connection uses the keepalive timer function, and remains open for a certain period of time after data packets are sent.

• TCP keepalive mechanism

  The keepalive mechanism is implemented using a timer. If the timer is activated, the server will send a keepalive probe packet. An ACK message is expected as a response. If the client does not respond, the server will terminate the connection. If the client responds, the keepalive timer will be reset.

  The keepalive duration on the server is set to 30m. In Linux, three parameters can be used to control the keepalive duration: tcp_keepalive_time (idle duration for enabling the keepalive function), tcp_keepalive_intvl (interval for sending keepalive packets), and tcp_keepalive_probes (the number of times the keepalive packets are sent if no response is received).

• http-keepalive

  The http-keepalive mechanism enables a TCP connection to transmit as many packets as possible. The http-keepalive duration is updated each time a packet is transmitted. If the http-keepalive duration expires, it indicates that the client and server did not exchange packets during this period. In this case, the connection is automatically closed and released.

  The tcp-keepalive mechanism retains a TCP connection until the connection is deliberately closed.