Memory Flow Control

Context

Elasticsearch provides a circuit breaker, which will terminate requests if the memory usage exceeds its threshold. However, Elasticsearch does not check the heap memory usage when an API is called, and does not allow users to configure the threshold for a single request. In this case, memory usage can only be calculated during request processing, which may lead to frequent circuit breaking and cannot avoid heap memory waste. To solve this problem, CSS checks the heap memory usage when receiving REST requests, blocking excess API requests and protecting nodes. You can configure global memory flow control, or configure the request path and heap memory threshold for a specific request path. Before a request is processed, the system checks the configured heap memory threshold. If the threshold is exceeded, the request path will be blocked.

• Memory flow control may affect request processing performance.
• If the memory flow control is enabled, some Kibana search requests may fail.
• If memory flow control is enabled in Elasticsearch 5.5.1, _mget requests will be blocked and Kibana access will be abnormal. You can add _mget requests to the request whitelist to avoid this problem.

The following table describes memory flow control parameters.

Procedure

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. On the Clusters page, locate the target cluster and click Access Kibana in the Operation column.
3. In the navigation pane on the left, choose Dev Tools and run commands to enable or disable memory flow control.
   • Enabling memory flow control

     PUT /_cluster/settings
     {
       "persistent": {
         "flowcontrol.memory.enabled": true,
         "flowcontrol.memory.allow_path": "/index/_search",
         "flowcontrol.memory.heap_limit": "85%"
       }
     }

   • Enabling memory flow control for a request path

     Configure the heap memory usage threshold for a request path. You can configure the priorities of such threshold rules.

     PUT /_cluster/settings
     {
       "persistent": {
         "flowcontrol.memory.enabled": true,
         "flowcontrol.memory": {
           "flowcontrol_search": {
             "filter_path": "index1/_search",
             "heap_limit": "50%"
           },
           "flowcontrol_bulk": {
             "filter_path": "index*/_bulk",
             "heap_limit": "50%"
           }
         }
       }
     }

   • Deleting the memory flow control configuration of a request path

     PUT /_cluster/settings
     {
       "persistent": {
         "flowcontrol.memory.enabled": true,
         "flowcontrol.memory": {
           "flowcontrol_search": {
             "filter_path": null,
             "heap_limit": null
           }
         }
       }
     }

   • Disabling cluster memory flow control

     PUT /_cluster/settings
     {
       "persistent": {
         "flowcontrol.memory.enabled": false
       }
     }