Failed to Write Data to a CSS Cluster Index

Symptom

Failed to write data to an Elasticsearch/OpenSearch cluster index. The following error message is returned:

"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"

The index has been set to read-only mode (read-only/allow delete), and all write operations (including creating and updating documents) are blocked.

Possible Cause

When the disk usage of the data node reaches 95%, Elasticsearch/OpenSearch automatically sets the index to read-only mode, that is, set index.blocks.read_only_allow_delete to true, to prevent system breakdown caused by disk space exhaustion. In this case, all write requests are rejected, and only the delete permission is reserved to release space.

The automatic recovery mechanism varies depending on the version:

• Elasticsearch 7.10.2 and OpenSearch: After the disk usage decreases, the read-only mode is automatically disabled.
• Versions earlier than Elasticsearch 7.10.2: After the disk usage decreases, you need to manually disable the read-only mode.

Solution

1. View the cluster monitoring metric to check the disk usage. If the disk usage reaches 95%, clear the disk or expand the disk capacity.
2. Clear the disk or expand the disk capacity.
   • Clear the disk: Delete indexes or expired data that is no longer needed.

     Run the following command to delete a specified index:

     DELETE /<index-name>

     

     Deleted indexes cannot be restored. Exercise caution.

   • Expand disk capacity: On the cluster management page, locate the target data node or cold data node, click More in the Operation column, and select Modify Configuration to expand the node's disk capacity.
3. Disable the read-only mode for the index.
   • Elasticsearch 7.10.2 and OpenSearch

     For these versions, the read-only mode is automatically disabled after the cluster disk usage decreases. Wait for about 1 to 2 minutes and try writing data again to check whether the fault is rectified.

   • Versions earlier than Elasticsearch 7.10.2

     For these versions, you need to manually disable the read-only mode after the cluster disk usage decreases.

     1. Run the following commands to disable the read-only mode for all indexes:

        PUT /_all/_settings
        {
          "index.blocks.read_only_allow_delete": null
        }

        

        Before running this command, ensure that the disk space is sufficient. Otherwise, read-only protection may be triggered again.

     2. Wait for about 1 to 2 minutes and try writing data again to check whether the fault is rectified.
4. To prevent the fault from re-occurring, you are advised to:
   • Periodically monitor the disk usage of the node and configure an alarm threshold (85% is recommended).
   • Periodically delete expired indexes and documents to release disk space.
   • Plan sufficient storage capacity and reserve 15% to 20% redundant disk space.