Configuring Automatic Retry for Failed Shard Allocation

How the Feature Works

After automatic retry for failed shard allocation is enabled, the system automatically retries shards whose number of allocation failures has reached the retry limit. The limit is controlled by index.allocation.max_retries and defaults to 5. The retry process is as follows:

1. Automatic retries start after the base delay.
2. If a retry succeeds, the shard is no longer tracked. If the retry fails, the backoff delay is doubled before the next retry.
3. Once the backoff delay reaches the configured maximum delay, it no longer increases. Subsequent retries are performed at the maximum delay.
4. The system periodically checks for failed shards that need to be retried at the configured polling interval. If any are found, they are retried.

Constraints

Automatic retry for failed shard allocation is supported only for Elasticsearch 7.10.2 clusters (image version 7.10.2_26.6.0_xxx or later). The feature is enabled by default for clusters that meet these version requirements.

Logging In to Kibana

Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Configuring Automatic Retry Parameters for Failed Shard Allocation

You can dynamically configure the parameters for automatic retry of failed shard allocation. The changes take effect immediately after configuration.

1. Enable or disable automatic retry for failed shard allocation.

   PUT _cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.retry_failed.auto_enabled": true
     }
   }

   Table 2 Parameter for enabling or disabling automatic retry for failed shard allocation

   Parameter	Type	Default Value	Description
   cluster.routing.allocation.retry_failed.auto_enabled	Boolean	true	Enables or disables automatic retry for failed shard allocation. The value can be: true: Enables automatic retry. false: Disables automatic retry. After automatic retry is disabled, shards whose number of allocation failures reaches the retry limit remain in the unassigned state and must be retried manually using the Reroute API, which may result in prolonged service interruption. It is recommended that you disable this feature only when automatic recovery is not required for your cluster.

2. Configure the exponential backoff parameters that control the retry interval and frequency.

   PUT _cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.retry_failed.auto_base_delay": "1m",
       "cluster.routing.allocation.retry_failed.auto_max_delay": "120m",
       "cluster.routing.allocation.retry_failed.auto_poll_interval": "5m"
     }
   }

   Table 3 Exponential backoff parameters

   Parameter	Type	Default Value	Description
   cluster.routing.allocation.retry_failed.auto_base_delay	TimeValue	1m	Base delay for the backoff strategy. This specifies the initial wait time before the first automatic retry. After each failed retry, the delay doubles. Value range: 1m–1440m Format: number + unit. Supported units: ms (milliseconds), s (seconds), m (minutes), h (hours), and d (days).
   cluster.routing.allocation.retry_failed.auto_max_delay	TimeValue	120m	Maximum delay for the backoff strategy. Once the retry interval reaches this value, it no longer increases, preventing the backoff delay from growing indefinitely. Value range: 1m–1440m Format: number + unit. Supported units: ms (milliseconds), s (seconds), m (minutes), h (hours), and d (days).
   cluster.routing.allocation.retry_failed.auto_poll_interval	TimeValue	5m	Polling interval. The system periodically checks for shards that have exceeded the retry limit, ensuring that shards are not missed when the cluster state remains stable. Value range: 1m–1440m Format: number + unit. Supported units: ms (milliseconds), s (seconds), m (minutes), h (hours), and d (days). It is recommended that you keep the default value. A smaller value increases the polling frequency and may have a slight impact on cluster performance. A larger value may delay the detection of shards that need to be retried.

3. (Optional) Verify the configuration. Query the current automatic retry configuration for failed shard allocation:

   GET _cluster/settings?filter_path=**.retry_failed.auto_*

   If the returned result contains the configured parameters and their values match the configured values, the configuration has taken effect.

FAQ

• Q1: How do I manually retry failed shard allocation after disabling automatic retry?

  After automatic retry is disabled, you can manually retry failed shard allocation by calling the Reroute API:

  POST /_cluster/reroute?retry_failed=true

• Q2: Does automatic retry for failed shard allocation affect ongoing shard allocation?

  No. Automatic retry only allocates shards that have reached the index.allocation.max_retries limit and are now in the unassigned state. It does not trigger Move or Balance operations and therefore does not affect ongoing shard allocation.