Help Center/ Cloud Search Service/ Troubleshooting/ Functions/ Troubleshooting Method for Logstash Pipeline Hot Stop Failure
Updated on 2026-08-21 GMT+08:00
View PDF
Share

Troubleshooting Method for Logstash Pipeline Hot Stop Failure

Symptom

A Logstash cluster writes data to a target Elasticsearch cluster using the logstash-output-elasticsearch plug-in. When updating the pipeline configuration using the hot stop feature, the hot stop fails, and the pipeline becomes blocked and unable to close properly. The following error message appears in the runtime logs: The shutdown process appears to be stalled due to busy or blocked plugins

Troubleshooting

Table 1 Possible causes and troubleshooting methods

No.

Probable Cause

Key Error Characteristics

Handling Method

1

Target index set to read-only

HTTP 403 + cluster_block_exception

For details, see Cause 1: Target Index Read-Only.

2

Target cluster JVM memory circuit breaker triggered

HTTP 429 + circuit_breaking_exception

For details, see Cause 2: Target Cluster JVM Memory Circuit Breaker.
  1. Log in to the CSS management console.
  2. In the navigation pane on the left, choose Clusters > Logstash.
  3. In the cluster list, click the target cluster name to enter the cluster details page.
  4. Click the Configuration Center tab and click Run Logs above the pipeline list.
  5. Search for error information in the runtime logs.
    • Typical error log example for target index read-only
      • Write phase: 
        [2026-07-31T08:30:38,426][ERROR][logstash.outputs.elasticsearch][pipeline.name][plugin.id] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"http://xx.xx.xx.xx:9200/_bulk"}
.
.
.
[2026-07-31T06:39:25,387][INFO ][logstash.outputs.elasticsearch][pipeline.name][plugin.id] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [target_index_name] blocked by: [FORBIDDEN/8/index write (api)];"})
      • Hot stop phase: 
        [2026-07-31T06:33:21,217][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
    • Typical error log example for target cluster JVM memory circuit breaker
      • Write phase: 
        [2026-07-31T08:30:38,426][ERROR][logstash.outputs.elasticsearch][pipeline.name][plugin.id] Encountered a retryable error. Will Retry with exponential backoff  {:code=>429, :url=>"http://xx.xx.xx.xx:9200/_bulk"}
.
.
.
[2026-07-31T06:39:25,387][INFO ][logstash.outputs.elasticsearch][pipeline.name][plugin.id] retrying failed action with response code: 429 ({"type"=>"circuit_breaking_exception", "reason"=>"fake circuit break here!!!!!!"bytes_wanted"=>0, "bytes_limit"=>0", "durability"=>"TRANSIENT"})
      • Hot stop phase: 
        [2026-07-31T08:31:13,951][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.

Cause 1: Target Index Read-Only

The index.blocks.write property of the target index is set to true. When Logstash writes data, it encounters a write block and receives an HTTP 403 rejection error. The logstash-output-elasticsearch plug-in treats the 403 error as a retryable error and continuously retries. Since the retry loop does not check the @stopping flag, the hot stop process waits for all in-flight events in the pipeline to complete before closing. However, events cannot complete due to continuous retries, causing the pipeline to block and time out.

Solution:
  1. Search for the 403 or cluster_block_exception keyword in the runtime logs.

    If the runtime logs contain HTTP 403 errors with cluster_block_exception, confirm this as cause 1 and proceed with the following steps.

  2. Force stop the pipeline.

    Stopping all pipelines will directly interrupt all running pipeline tasks in the cluster. Data in transit may experience brief delays or offset resets. Ensure that the source data is traceable.

    1. On the Configuration Center tab, click Stop All above the pipeline list.
    2. In the displayed dialog box, click OK.
    3. Check the Status column in the pipeline list to confirm that all pipelines display a status of Stopped.
  3. Remove the read-only status of the target index.

    Log in to the Kibana page of the target Elasticsearch cluster. In *Dev Tools, run the following command to remove the index read-only status:

    PUT <index-name>/_settings
{
  "index.blocks.write": false
}
  4. Restart the pipeline tasks.
    1. In the configuration file list on the Configuration Center tab, select the configuration files to start and click Start Logstash above.
    2. In the displayed dialog box, confirm the persistent configuration and click OK.
    3. In the pipeline list, confirm that the target pipeline's Status displays Running and the Events column data is continuously updating, indicating that the data migration task has resumed.

Cause 2: Target Cluster JVM Memory Circuit Breaker

The JVM memory usage of the target Elasticsearch cluster exceeds 95%, triggering the parent-level circuit breaker mechanism and returning a BadResponseCodeError 429 error. Similar to cause 1, the logstash-output-elasticsearch plug-in treats the 429 error as a retryable error and continuously retries. This causes the hot stop process to wait for in-flight events to complete, resulting in pipeline blocking and timeout.

Solution:
  1. Search for the 429 or circuit_breaking_exception keyword in the runtime logs.

    If the runtime logs contain HTTP 429 errors with circuit_breaking_exception, confirm this as cause 2 and proceed with the following steps.

  2. Force stop the pipeline.

    Stopping all pipelines will directly interrupt all running pipeline tasks in the cluster. Data in transit may experience brief delays or offset resets. Ensure that the source data is traceable.

    1. On the Configuration Center tab, click Stop All above the pipeline list.
    2. In the displayed dialog box, click OK.
    3. Check the Status column in the pipeline list to confirm that all pipelines display a status of Stopped.
  3. Restore JVM memory in the target cluster.

    Reduce the JVM memory usage of the target Elasticsearch cluster using the following methods:

    • Reduce write pressure: Decrease the number of concurrent write requests and reduce the batch write size.
    • Release memory: Run the following command in Kibana to trigger cache clearing: 
      POST /_cache/clear
    • Scale out/up the cluster: Increase the number of nodes in the target cluster or upgrade node specifications to increase JVM memory capacity.
  4. Restart the pipeline tasks.
    1. In the configuration file list on the Configuration Center tab, select the configuration files to start and click Start Logstash above.
    2. In the displayed dialog box, confirm the persistent configuration and click OK.
    3. In the pipeline list, confirm that the target pipeline's Status displays Running and the Events column data is continuously updating, indicating that the data migration task has resumed.

Submitting a Service Ticket

If the problem persists, submit a service ticket.

Parent topic: Functions