Creating a Vector Index

Logging In to Kibana

Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the displayed cluster list, find the target cluster, and click Access Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

Creating a Vector Index

Run the following command on Kibana to create a vector index.

For example, create an index named my_index. This index contains a vector field named my_vector and a text field named my_label. A graph-based index is created for the vector field, and Euclidean distance is used for similarity measurement.

PUT my_index 
{
  "settings": {
    "index": {
      "vector": true,
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "vector",
        "dimension": 2,
        "indexing": true,
        "algorithm": "GRAPH",
        "metric": "euclidean"
      },
      "my_label": {
        "type": "keyword"
      }
    }
  }
}

(Optional) Pre-Building and Registering Centroid Vectors

When the IVF_GRAPH or IVF_GRAPH_PQ algorithm is used for vector indexing, you need to pre-build and register centroid vectors.

IVF_GRAPH and IVF_GRAPH_PQ help to accelerate indexing and queries in ultra-large-scale clusters. They allow you to narrow down the query scope by dividing a vector space into subspaces through clustering or random sampling. Before pre-build, you need to obtain all centroid vectors through clustering or random sampling. Centroid vectors are pre-built into a GRAPH or GRAPH_PQ index and then registered with the Elasticsearch cluster. All nodes in the cluster can share this index. Reuse of the centroid index among shards can effectively reduce the training overhead and the number of centroid index queries, improving write and query performance.

1. Create a centroid index.

   For example, run the following command on Kibana to create a centroid index named my_dict:

   PUT my_dict 
    { 
      "settings": { 
        "index": { 
          "vector": true 
        }, 
        "number_of_shards": 1, 
        "number_of_replicas": 0 
      }, 
      "mappings": { 
        "properties": { 
          "my_vector": { 
            "type": "vector", 
            "dimension": 2, 
            "indexing": true, 
            "algorithm": "GRAPH", 
            "metric": "euclidean" 
          } 
        } 
      } 
    }

   For detailed parameter configuration, see Creating a Vector Index. Pay attention to the following mandatory parameters:

   • index.number_of_shards: The number of index shards must be set to 1. Otherwise, the centroid index cannot be registered.
   • indexing: This parameter must be set to true to enable vector index acceleration.
   • algorithm: Set the indexing algorithm. Set it to GRAPH for the IVF_GRAPH algorithm, and GRAPH_PQ if the IVF_GRAPH_PQ algorithm is used.
2. Write centroid vectors to the created index. Write the centroid vectors obtained through sampling or clustering into the newly created index my_dict.
3. Call the registration API.

   Run the following command on Kibana to register the centroid index as a Dict object with a globally unique name (dict_name):

   PUT _vector/register/my_dict 
    { 
      "dict_name": "my_dict" 
    }

4. Create an IVF_GRAPH or IVF_GRAPH_PQ vector index.

   When creating the vector index, you do not need to specify dimension or metric. Rather, you specify the registered Dict object. Table 5 describes key parameters for specifying a Dict object. For details about other parameters, see Creating a Vector Index.

   For example, run the following command to create an IVF_GRAPH vector index:

   PUT my_index 
    { 
      "settings": { 
        "index": { 
          "vector": true,
          "sort.field": "my_vector.centroid" # Set the centroid subfield of each vector field as a ranking field.
        } 
      }, 
      "mappings": { 
        "properties": { 
          "my_vector": { 
            "type": "vector", 
            "indexing": true, 
            "algorithm": "IVF_GRAPH", 
            "dict_name": "my_dict", 
            "offload_ivf": true 
          } 
        } 
      } 
    }

   Table 5 Field mappings parameters

   Parameter	Mandatory	Type	Description
   dict_name	Yes	String	Name of the centroid index. For example, my_dict. The vector dimensions and metrics of the index must be the same as those of the Dict index.
   offload_ivf	Yes	Boolean	Whether to offload the IVF inverted index to Elasticsearch. Value: true or false. true (recommended value): Offloads the IVF inverted index implemented by the underlying index to Elasticsearch. This reduces the use of off-heap memory and the overhead of write and merge operations. false (default value): Not to offload the IVF inverted index.