A few weeks ago, the Elasticsearch node that indexes products for one of our clients stopped updating documents. After a few hours of searching, I learned Elasticsearch disables updating indexes when a certain disk space threshold is reached. I thought it a good idea to write about it, so you might be able to use this solution if you encounter the same problem.
The first sign that something was wrong was a call from the client that new products were not appearing on their website yet perfectly visible in their admin panel. I didn’t take long to figure out that whenever a document is updated Elasticsearch returned the following errors:
[2021-02-25T13:42:30][INFO] low disk watermark [85%] exceeded on [94UY8h3x3wt2SAZSvjkGt1][products]
[2021-02-25T13:42:43][INFO] low disk watermark [85%] exceeded on [94UY8h3x3wt2SAZSvjkGt1][products]
The error made it clear to me that something was wrong with the disk. But at the time, I had no clue how this could impact Elasticsearch.
By default, Elasticsearch requires the server to have 85% of its disk capacity left to update documents on its local indexes. In my case, the total size of the disk in the machine was 1,2TB of data, meaning there was about 180 gigabytes of storage left which is plenty.
Elasticsearch has several configurations that manage locks based on disk space, but the most important one for us is cluster.routing.allocation.disk.watermark.low. This key indicates at what stage the local cluster should lock all indexes for writing. The value of this key can be set either to a percentage or a static value such as "500MB".
Because my server has more than enough disk space I opted to set the low barrier to a higher percentage. The configuration for the local Elasticsearch can easily be updated using a curl call like so:
curl -XPUT 'http://localhost:9200/_cluster/settings' \
-H 'Content-Type: application/json' \
-d '{"transient" : {"cluster.routing.allocation.disk.watermark.low" : "95%"}}'
Updating the cluster settings will not yet fix the index lock and you will still not be able to update documents in the index. I had to learn this the hard way as I did not test reindexing after the settings solution.
When Elasticsearch detects the disk is filling up, it will set a flag on each index individually telling it to go in read-only mode. The index setting index.blocks.read_only_allow_delete will be set to true. Updating the watermark setting will not automatically reset this flag, this is something you will have to do manually.
Using another curl call, we can reset this flag in the settings for the specific index. In the case below, products is the name of the index we are updating.
curl -XPUT 'http://localhost:9200/products/_settings' \
-H 'Content-Type: application/json' \
-d '{"index.blocks.read_only_allow_delete": null}'