Help Center/ Cloud Search Service/ Best Practices/ Elasticsearch Vector Search/ Using the IVF_GRAPH_PQ Algorithm to Implement Vector Search
Updated on 2025-09-05 GMT+08:00

Using the IVF_GRAPH_PQ Algorithm to Implement Vector Search

By combining shared, pre-built centroid indexes (GRAPH_PQ) with IVF_GRAPH_PQ vector indexes, CSS vector databases enable high-performance and low-cost search through tens of billions of vector records.

Scenario

IVF_GRAPH_PQ is a composite indexing algorithm. Its core components include:

  • IVF (Inverted File): generates centroid vectors through clustering, thus partitioning the dataset into clusters by centroid. This prunes the search space in graph indexes and enhances query efficiency.
  • PQ (Product Quantization): quantizes and compresses high-dimensional vectors to reduce storage and computing overheads.
  • Centroid sharing: Multiple shards or segments reuse the same set of centroids, avoiding redundant clustering computations.

Vector databases that use the IVF_GRAPH_PQ algorithm are designed to handle large-scale vector search, supporting datasets exceeding 10 billion records with high-throughput ingestion (e.g., 100 million new vectors daily). Examples of such applications include image recognition, recommendation systems, and natural language processing, which all involve vector similarity search.

Solution Procedure

  1. Generate centroid vectors through clustering or sampling and create a GRAPH_PQ centroid index.
  2. Use the IVF_GRAPH_PQ algorithm to create a vector index and associate it with the centroid index.
  3. Ingest vector data.
  4. Enable efficient vector similarity search.

Highlights

  • Centroid sharing: Indexes share centroid vectors across shards or segments, avoiding redundant clustering computations.
  • Graph index acceleration: Search space is reduced for graph indexes, thus enhancing query performance.
  • Cost reduction with PQ: Index storage, query, and computation overheads are reduced.
  • Resource optimization: Indexes do not require resident memory, cutting resource costs.

Prerequisites

  • A CSS Elasticsearch vector database has been created. The cluster version is 7.10.2, and the cluster node flavor is memory-optimized.
  • Centroid vector data (accounting for 0.001% to 0.01% of the total data size) has been obtained through clustering or random sampling.

Procedure

  1. Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana integrated by CSS as an example to describe the operation procedures.

    1. Log in to the CSS management consoleCSS management console.
    2. In the navigation pane on the left, choose Clusters > Elasticsearch.
    3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
    4. In the left navigation pane, choose Dev Tools.

  2. Create a centroid index.

    Run the following command to create a centroid index, for example, my_dict.

    number_of_shards must be set to 1, and algorithm must be set to GRAPH_PQ.
    PUT my_dict 
    { 
      "settings": { 
        "index": { 
          "vector": true 
        }, 
        "number_of_shards": 1, 
        "number_of_replicas": 0 
      }, 
      "mappings": { 
        "properties": { 
          "my_vector": { 
            "type": "vector", 
            "dimension": 4, 
            "indexing": true, 
            "algorithm": "GRAPH_PQ", 
            "metric": "euclidean" 
          } 
        } 
      } 
    }

  3. Import centroid vectors.

    Write the centroid vectors obtained through sampling or clustering into the newly created centroid index.

    • Write a single record:
      POST my_dict/_doc
      {
        "my_vector": [1.0, 1.0, 1.0, 1.0]
      }
    • Write multiple records at the same time.
      POST my_dict/_bulk
      {"index": {}}
      {"my_vector": [2.0, 2.0, 2.0, 2.0]}
      {"index": {}}
      {"my_vector": [3.0, 3.0, 3.0, 3.0]}
      {"index": {}}
      {"my_vector": [4.0, 4.0, 4.0, 4.0]}

  4. Register the centroid index.

    Run the following command to register the centroid index as a Dict object with a globally unique name (dict_name). dict_name is configurable.

    PUT _vector/register/my_dict 
    { 
      "dict_name": "my_dict" 
    }

  5. Create an IVF_GRAPH_PQ index.

    Run the following command to create an IVF_GRAPH_PQ vector index (for example, my_index) and associate it with the centroid index registered in the previous step through dict_name.

    PUT my_index 
    { 
      "settings": { 
        "index": { 
          "vector": true,
          "sort.field": "my_vector.centroid"  # Set the centroid subfield of each vector field as a ranking field.
        } 
      }, 
      "mappings": { 
        "properties": { 
          "my_vector": { 
            "type": "vector", 
            "indexing": true, 
            "algorithm": "IVF_GRAPH_PQ", 
            "dict_name": "my_dict", 
            "offload_ivf": true 
          } 
        } 
      } 
    }

  6. Import the full vector data.

    Run the following command to write the full vector data to the new IVF_GRAPH_PQ index.

    • Write a single record:
      POST my_index/_doc
      {
        "my_vector": [2.0, 3.0, 4.0, 5.0]
      }
    • Write multiple records at the same time:
      POST my_index/_bulk
      {"index": {}}
      {"my_vector": [2.0, 3.0, 4.0, 5.0]}
      {"index": {}}
      {"my_vector": [1.0, 2.0, 3.0, 4.0]}
      {"index": {}}
      {"my_vector": [3.0, 4.0, 5.0, 6.0]}

  7. Query vector data.

    Run the following command to perform a vector search:

    POST my_index/_search
    {
      "size":2,
      "query": {
        "vector": {
          "my_vector": {
            "vector": [1.0, 1.0, 1.0, 1.0],
            "topk":2
          }
        }
      }
    }

    If the two most relevant results are returned, the query is successful.

Related Documents