Usage of IVF_GRAPH and IVF_GRAPH_PQ Algorithms

Scenario

In the vector index acceleration algorithms, IVF_GRAPH and IVF_GRAPH_PQ apply to scenarios with a large amount of data. These two algorithms allow you to narrow down the query range by dividing a vector space into subspaces through clustering or random sampling. Before narrowing down the query range, you need to complete clustering or random sampling to generate a center point vector in the subspace.

Then, pre-create and register the center point vector to create the GRAPH or GRAPH_PQ index and register it with the Elasticsearch cluster so that the index file can be shared among multiple nodes. Reuse of the center index among shards can effectively reduce the training overhead and the number of center index queries, improving the write and query performance.

Procedure

Create a center point index table.

For example, if the created index is named my_dict, number_of_shards of the index must be set to 1. Otherwise, the index cannot be registered.

To use the IVF_GRAPH index, set algorithm of the center point index to GRAPH.

To use the IVF_GRAPH_PQ index, set algorithm of the center point index to GRAPH_PQ.

PUT my_dict 
 { 
   "settings": { 
     "index": { 
       "vector": true 
     }, 
     "number_of_shards": 1, 
     "number_of_replicas": 0 
   }, 
   "mappings": { 
     "properties": { 
       "my_vector": { 
         "type": "vector", 
         "dimension": 2, 
         "indexing": true, 
         "algorithm": "GRAPH", 
         "metric": "euclidean" 
       } 
     } 
   } 
 }

Writes the center point vector to the created index. Write the center point vector obtained through sampling or clustering into the created my_dict index by referring to Importing a Vector.
Call the registration API.

Register the created my_dict index with a Dict object with a globally unique identifier name (dict_name).
```
PUT _vector/register/my_dict?timout=120s
 {
  "dict_name": "my_dict"
 }
```
To prevent registration timeout caused by a large dictionary, the default value of timeout is 10 minutes. You can change the value.

Create an IVF_GRAPH or IVF_GRAPH_PQ index.

When you create the IVF_GRAPH or IVF_GRAPH_PQ index, you only need to specify the name of the registered Dict.

PUT my_index 
 { 
   "settings": { 
     "index": { 
       "vector": true 
     } 
   }, 
   "mappings": { 
     "properties": { 
       "my_vector": { 
         "type": "vector", 
         "indexing": true, 
         "algorithm": "IVF_GRAPH", 
         "dict_name": "my_dict", 
         "offload_ivf": false 
       } 
     } 
   } 
 }

**Table 1** Field mappings parameter description
Parameter	Remarks
dict_name	Specifies the name of the depended center point index. The vector dimension and measurement metric of the index are the same as those of the Dict index.
offload_ivf	Unloads the IVF inverted index implemented by the underlying index to Elasticsearch. In this way, the use of non-heap memory and write or merge operation overhead are reduced. However, the query performance also deteriorates. Generally, the default value is used. The value can be true or false. Default value: false