Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Ankara Region)/ Using Elasticsearch/ In-House Plug-Ins/ Vector Search/ Configuration of Other Parameters
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Configuration of Other Parameters

Index Configurations

The vector search plug-in integrates the developed engine and provides index acceleration algorithms such as GRAPH. It provides some index parameters to achieve higher query performance or precision when vector indexes are used.

  • GRAPH parameters
    • neighbors: number of neighbors of each vector in a graph index. The default value is 64. A larger value indicates higher query precision. The larger the index, the slower the construction and query speed.
    • shrink: cropping coefficient during HNSW construction. The default value is 1.0f.
    • scaling: scaling multiple of the data volume of the HNSW upper- and lower-layer graphs. The default value is 50.
    • efc: queue size of the neighboring node when the HNSW is constructed. The default value is 200. A larger value indicates higher precision and slower construction speed.
    • max_scan_num: maximum number of nodes to be scanned. The default value is 10000. A larger value indicates higher precision and slower index speed.
  • PQ parameters for creating a GRAPH_PQ index or a PQ codebook
    • centroid_num: number of clustering center points of each fragment. The default value is 255.
    • fragment_num: number of fragments. The default value is 0. The plug-in automatically sets the number of segments based on the vector length.

Querying Configurations

In addition to index parameters, the plug-in also provides query parameters for vector indexes. You can adjust the GRAPH vector index parameters to achieve higher query performance or precision.

GRAPH parameters
  • ef: queue size of the neighbor node during the query. A larger value indicates higher query precision and slower query speed. The default value is 200.
  • max_scan_num: maximum number of nodes to be scanned. A larger value indicates higher precision and slower query speed. The default value is 10000.

Cluster Configurations

For details about the cluster configuration parameters, see Table 1.

Table 1 Parameters for configuring a cluster

Parameter

Remarks

native.cache.circuit_breaker.enabled

Whether to enable the non-heap memory circuit breaker. The default value is true.

native.cache.circuit_breaker.cpu.limit

Upper usage of non-heap memory that can be used by the index vector. The default value is 45%.

For example, if the overall memory of a host is 128 GB and the heap memory occupies 31 GB, the default upper limit of the non-heap memory usage is (128 - 31) * 45% = 43.65 GB. If the non-heap memory usage exceeds the upper limit, the circuit breaker is triggered.

native.cache.expire.enabled

Whether to enable the cache expiration time. The default value is false. Possible values are as follows:

  • true: Enabled. When this function is enabled, cache items that have not been accessed for a long time will be cleared.
  • false: Disabled.

native.cache.expire.time

Timeout duration. The default value is 24h.

native.vector.index_threads

Number of threads used when the underlying index is created. You are not advised to set this parameter to a large value. Otherwise, the overall write performance will be affected. The default value is 4.

path.vector.data

Path for storing the vector retrieval dictionary file. The default value is /srv/BigData/elasticsearch/{Instance name}/vector.
  • In offline data import scenarios, to improve the performance of batch write operations, you are advised to set the refresh_interval parameter of the index to -1 to disable automatic index refreshing.
  • You are advised to set number_of_replicas to 0. After the offline data is imported, set it to the required value.

Index Cache Query

The vector search engine is implemented using C++ and occupies the non-heap memory. Two APIs are provided by this engine to better manage the index cache.

  • Viewing the cache

    In the implementation of the vector plug-in, the vector index is the same as other types of Lucene indexes. Each segment constructs and stores an index file. During query, the index file is loaded to the non-heap memory. The plug-in uses the cache mechanism to manage the non-heap memory. You can use the provided APIs to query the non-heap memory usage, number of cache hits, and number of loading times.

  • Clearing the cache

    PUT /_vector/clear/cache

    PUT /_vector/clear/cache/{index}

    The caching mechanism limits the non-heap memory usage when vector indexes are used. When the total index size exceeds the cache size limit, index entry swap-in and swap-out occur, which affects the query performance. You can call the target API to clear the index cache that is no longer used to ensure the query performance of hot data indexes.

Vector Index Query

  • View vector retrieval statistics.

    GET /_vector/stats

  • Preload the vector index.

    PUT /_vector/warmup/{index}

Vector Retrieval Dictionary

  • Register a dictionary.

    PUT /_vector/register/{index}

    {

    "dict_name": "my_dict"

    }

  • List vector retrieval dictionaries.

    GET /_vector/list/{type}

  • Deregister a dictionary.

    PUT /_vector/unregister

    {

    "dict_name": "my_dict"

    }

    Deregistering a dictionary is risky. If an IVF index is still using the dictionary after the deregistration, the index will become abnormal in scenarios such as restart because the dictionary is unavailable. Exercise caution when performing this operation.

Parent topic: Vector Search