Using Vector Indexes for Data Search in an Elasticsearch Cluster

Standard Query

Standard vector query syntax is provided for vector fields with vector indexes. The following command will return n (specified by size/topk) data records that are most close to the query vector.

POST my_index/_search
{
  "size":2,
  "_source": false, 
  "query": {
    "vector": {
      "my_vector": {
        "vector": [1, 1],
        "topk":2
      }
    }
  }
}

Composite Query

Vector search can be used with other Elasticsearch subqueries to implement a composite query, such as pre-filtering query and Boolean query.

Only Elasticsearch 7.10.2 clusters support pre-filtering queries.

In the following example, the top 10 records whose my_label value is red are queried.

• Pre-filtering query: First, filters are used to retrieve matching results. Then, vector search is performed on these results to find the most relevant vectors based on similarity.
  The following is an example of the code:

  POST my_index/_search
  {
    "size": 10,
    "query": {
      "vector": {
        "my_vector": {
          "vector": [1, 2],
          "topk": 10,
          "filter": {
            "term": { "my_label": "red" }
          }
        }
      }
    }
  }

  

  If the filter is too restrictive, leading to a small intermediate result set, you can set the index parameter index.vector.exact_search_threshold to switch from pre-filtering query to brute-force query, thus improving the recall of the query. For details about the parameter, see Creating a Vector Index.

• Boolean query: A Boolean query is in fact a post-filtering query method. Filtering and vector similarity-search are performed separately. Then, the results of the two are combined. The logic of combination is determined by clauses like must, should, and filter.
  The following is an example of the code:

  POST my_index/_search
  {
    "size": 10,
    "query": {
      "bool": {
        "must": {
          "vector": {
            "my_vector": {
              "vector": [1, 2],
              "topk": 10
            }
          }
        },
        "filter": {
          "term": { "my_label": "red" }
        }
      }
    }
  }

ScriptScore Query

You can use script_score to perform Nearest Neighbor Search (NSS) on vectors. The query syntax is provided below.

The pre-filtering condition can be any query. script_score traverses only the pre-filtered results, calculates the vector similarity, and sorts and returns the results. This query method does not use vector indexes. Its performance depends on the size of the intermediate result set after the pre-filtering. If the pre-filtering condition is set to match_all, brute-force search is performed on all data.

POST my_index/_search 
 { 
   "size":2, 
   "query": { 
   "script_score": { 
       "query": { 
         "match_all": {} 
       }, 
       "script": { 
         "source": "vector_score", 
         "lang": "vector", 
         "params": { 
           "field": "my_vector", 
           "vector": [1.0, 2.0], 
           "metric": "euclidean" 
         } 
       } 
     } 
   } 
 }

Re-Score Query

If the GRAPH_PQ or IVF_GRAPH_PQ index is used, the query results are sorted based on the asymmetric distance calculated by PQ. CSS supports re-scoring and ranking of query results to improve the recall rate.

Assuming that my_index is a PQ index, an example of re-scoring the query results is as follows:

GET my_index/_search 
 { 
   "size": 10, 
   "query": { 
     "vector": { 
       "my_vector": { 
         "vector": [1.0, 2.0], 
         "topk": 100 
       } 
     } 
   }, 
   "rescore": { 
     "window_size": 100, 
     "vector_rescore": { 
       "field": "my_vector", 
       "vector": [1.0, 2.0], 
       "metric": "euclidean" 
     } 
   } 
 }

Painless Syntax Extension

CSS extension supports multiple vector distance calculation functions, which can be directly used in customized painless scripts to build flexible re-score formulas.

The following is an example:

POST my_index/_search
{
  "size": 10,
  "query": {
    "script_score": {
      "query": {
        "match_all": {}
      },
      "script": {
        "source": "1 / (1 + euclidean(params.vector, doc[params.field]))",
        "params": {
          "field": "my_vector",
          "vector": [1, 2]
        }
      }
    }
  }
}