Enhanced Cold Data Query Performance

Context

When you query data on the Discover page of Kibana for the first time, all data needs to be obtained from OBS because there is no cache. If a large number of documents are returned, it takes a long time to obtain the corresponding time fields and file metadata from OBS. To accelerate queries the first time they run on the Discover page, you can cache data locally.

Prerequisites

This feature is available in Elasticsearch clusters of versions 7.6.2 and 7.10.2 and Opensearch clusters created after February 2023.

API for Querying Cold Data from Local Cache

This API can be used to query the cold data from local cache.

Example request:

GET /_frozen_stats/local_cache
GET /_frozen_stats/local_cache/{nodeId}

Response example:

{
   "_nodes" : {
     "total" : 1,
     "successful" : 1,
     "failed" : 0
   },
   "cluster_name" : "elasticsearch",
   "nodes" : {
     "6by3lPy1R3m55Dcq3liK8Q" : {
       "name" : "node-1",
       "transport_address" : "127.0.0.1:9300",
       "host" : "127.0.0.1",
       "ip" : "127.0.0.1",
       "local_cache" : {
         "get_stats" : {
           "get_total_count" : 562,                            //Total number of times data was retrieved from the local cold data cache.
           "get_hit_count" : 562,                              //Total number of hits in the local cold data cache.
           "get_miss_count" : 0,                               //Total number of local cold data cache misses.
           "get_total_ns" : 43849200,                          //Total duration for retrieving data from the local cold data cache.
           "get_avg_ns" : 78023                                //Average duration for retrieving data from the local cold data cache.
         },
         "load_stats" : {
           "load_count" : 2,                                    //Number of times cold data was loaded from the local cache
           "load_total_ms" : 29,                                //Total duration for loading cold data from the local cache
           "load_avg_ms" : 14,                                  //Average duration for loading cold data from the local cache
           "load_fail_count" : 0,                               //Number of failure times for loading cold data from the local cache
           "load_overflow_count" : 0                            //Number of times the local cold data cache exceeds the cache pool size.
         },
         "reload_stats" : {
           "reload_count" : 0,                                  //Number of times the local cold data cache was regenerated.
           "reload_total_ms" : 0,                               //Total duration for regenerating the local cold data cache.
           "reload_avg_ms" : 0,                                 //Average duration for regenerating the local cold data cache.
           "reload_fail_count" : 0                              //Number of failures in regenerating the local cold data cache.
         },
         "init_stats" : {
           "init_count" : 0,                                     //Number of times the local cold data cache was initialized.
           "init_total_ms" : 0,                                  //Total duration for initializing the local cold data cache.
           "init_avg_ms" : 0,                                    //Average duration for initializing the local cold data cache.
           "init_fail_count" : 0                                 //Number of failures in initializing the local cold data cache.
         }
       }
     }
   }
 }

Configuring Parameters

Modifying Parameters

• Run the following command to modify low_cost.local_cache.max.capacity:

  PUT _cluster/settings
   {
     "persistent": {
       "low_cost.local_cache.max.capacity":1000
     }
   }

• Run the following command to modify index.low_cost.local_cache.threshold:

  PUT es_write_pref2-00000000021/_settings
   {
   "index.low_cost.local_cache.threshold":20
   }

• Run the following command to modify index.low_cost.local_cache.evict_time:

  PUT es_write_pref2-00000000021/_settings
   {
   "index.low_cost.local_cache.evict_time":"7d"
   }