A Cluster Is Stuck in the Creating Snapshot State

If a cluster's task status is stuck in Creating snapshot, there are two possible causes:

1. The cluster is under heavy load or contains large volumes of data, and snapshots take longer to create

   When a cluster is under heavy load (indexing or query) or contains large volumes of data, snapshots take significantly longer time to create. By default, both Elasticsearch and OpenSearch set an upper limit on the per-node snapshot speed (typically 40 MB/s). Additionally, the overall snapshot creation speed depends on the cluster resource availability. When under heavy load, the snapshot creation process takes longer.

   Solution:

   1. Query the snapshot progress: Use an API to check the snapshot progress and estimate the remaining time.

      GET _snapshot/repo_auto/{snapshot_name}

      Replace {snapshot_name} with the actual snapshot name.

      In the response, check the shards_stats field (or a similar field, depending on the version) to learn the number of completed shards and the total number of shards.

   2. Choose a solution.
      • Wait: If the estimated remaining time is acceptable or the cluster load is expected to decrease, you can wait for the snapshot to complete.
      • Terminate: If the estimated remaining time is too long, call the snapshot deletion API to terminate the task.

        DELETE _snapshot/repo_auto/{snapshot_name}

2. The snapshot status has failed to be updated

   Both Elasticsearch and OpenSearch save the status information of ongoing snapshots in a place called cluster state. After a snapshot is created, the task status needs to be updated in cluster state. If the update fails (for example, the request for updating the cluster state is rejected by the thread pool or has expired due to high memory usage in the cluster), the snapshot status in cluster state will be stuck in the Creating state.

   Solution:

   Call the snapshot deletion API to delete the snapshot task. This will clear the residual status.

   DELETE _snapshot/repo_auto/{snapshot_name}