Configuring the Kibana Query Response Circuit Breaker

How the Feature Works

When the Kibana backend process receives a query response, the circuit breaker determines whether to allow or block the response before buffering the response body.

Figure 1 Query process

The circuit breaker is triggered when either of the following conditions is met. After it is triggered, the response body is discarded directly:

• Absolute response size limit: The circuit breaker is triggered when maxContentLength > 0 and the response body's contentLength ≥ maxContentLength. This indicates that the response body itself is too large and is rejected regardless of the current heap usage.
• Heap usage limit: The circuit breaker is triggered when contentLength × heapMultiplier + heapUsed > heapSizeLimit × maxPercentage. This indicates that buffering the response is estimated to exceed the configured heap usage limit.

Key parameters are as follows:

• maxContentLength: Absolute response body size limit. The circuit breaker is triggered when the response body reaches this size.
• contentLength: Size of the current response body, which is the value of the content-length field in the HTTP response header.
• heapSizeLimit: Maximum heap size of the process.
• heapUsed: Current heap usage of the process.
• heapMultiplier: Heap usage estimation multiplier. The default value is 4. The value of contentLength is multiplied by this value to estimate the heap usage after loading the response into memory. The default value of 4 is used because JSON data parsed into JavaScript objects typically consumes approximately 3 to 5 times more memory than the original text. A multiplier of 4 provides a balance between protection and resource utilization.
• maxPercentage: Maximum allowed heap usage in percentage.

The query response circuit breaker applies to the Kibana backend process and complements the Elasticsearch cluster feature Configuring Large Query Isolation. The former protects the Kibana process from oversized responses, while the latter manages large queries on the Elasticsearch engine side.

Constraints

• The Kibana query response circuit breaker is supported only for Elasticsearch 7.10.2 clusters (image version 7.10.2_26.6.0_xxx or later). It is enabled by default for clusters that meet these version requirements.
• For clusters with the security mode enabled, Kibana query response circuit breaker parameters are visible and configurable only in the Global tenant.
• Changes made in Advanced Settings do not take effect immediately. They are synchronized periodically in the background and take effect within 1 minute.

Logging In to the Kibana Advanced Settings Page

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. (Optional) For a security-mode cluster, switch to the Global tenant before configuring Kibana query response circuit breaker parameters.

   On the Kibana page, click the username in the upper right corner and choose Switch tenants. On the Select your tenant page, select Global, and click Confirm.

5. In the left navigation pane, choose Management > Stack Management, and select Advanced settings.

Configuring Kibana Query Response Circuit Breaker Parameters

1. On the Settings page, locate the Search module and configure the following Kibana query response circuit breaker parameters.
   Figure 2 Configuring the Kibana query response circuit breaker
   

   Table 1 Configuring Kibana query response circuit breaker parameters

   Parameter	Type	Default Value	Description
   circuitBreaker:enabled	Boolean	On	Whether to enable the Kibana query response circuit breaker. On: Enables the circuit breaker. Off: Disables the circuit breaker. After the circuit breaker is disabled, the Kibana backend process no longer blocks oversized responses. Large query responses may cause the process to crash due to OOM, affecting all users. Disable this feature only when there is no risk of large queries in your workloads.
   circuitBreaker:heapMultiplier	Integer	4	Heap usage estimation multiplier used to estimate heap consumption after a response is loaded into the heap. Value range: 1–10
   circuitBreaker:maxContentLength	Integer	104857600	Absolute response body size limit. Value range: 0 (no limit) 1048576 (1 MB) to 1073741824 (1 GB) Unit: bytes
   circuitBreaker:maxPercentage	Float	1.0	Maximum allowed heap usage in percentage. Value range: 0.1–1.0 1.0 indicates that the entire available heap can be used.

2. Click Save changes to save the changes.
3. Wait for about 1 minute. Then run a large query to verify that the circuit breaker takes effect.
   1. On Kibana, expand the menu in the upper-left corner, and choose Dev Tools.
   2. Run a large query. The request fails with HTTP status code 502.
      Figure 3 Large query
      
   3. In the browser, press F12 to launch Developer Tools, switch to the Network tab, and check Response. If Memory circuit breaker triggered is displayed, the query response has been blocked by the circuit breaker.
      Figure 4 Query response