Creating an Elasticsearch Vector Cluster

Memory Planning

Before creating an Elasticsearch vector cluster, properly plan the cluster's memory capacity based on the data size, vector dimensions, and index types.

• Node type

  The CSS vector search engine relies heavily on in-memory computing, and vector indexing relies on off-heap memory. For a vector cluster, we recommend memory-optimized nodes.

• Memory calculation formula

  Estimate the required off-heap memory based on the index algorithms used, vector dimensions, and data volume.

  • FLAT index: mem_size = dim x dim_size x num + delta
  • GRAPH index: mem_size = (dim x dim_size + neighbors x 4) x num + delta
  • GRAPH_PQ index: mem_size = (fragment_num + neighbors x 4) x num + delta
  • GRAPH_SQ8 index: mem_size = (dim x 2 + neighbors x 4) x num + delta
  • GRAPH_SQ4 index: mem_size = (dim + neighbors x 4) x num + delta
  • IVF_GRAPH and IVF_GRAPH_PQ indexes do not require resident memory, so there is need to estimate their memory requirements.

  Table 1 Parameter description

  Parameter	Description
  mem_size	Size of off-heap memory required by vector indexes. (This estimate assumes there are no replicas. If index replicas are involved, the required memory size multiplies with the number of replicas.)
  dim	Number of vector dimensions.
  dim_size	Number of bytes required by each dimension. By default, each dimension is a 4-byte float value.
  neighbors	Number of neighbors for each vector in a graph index. The default value is 64.
  num	Total number of vectors.
  fragment_num	Number of segments a vector is split into when product quantization is used. If fragment_num is not configured during index creation, the value is determined by dim. if dim <= 256: fragment_num = dim / 4 elif dim <= 512: fragment_num = dim / 8 else : fragment_num = 64
  delta	Metadata size. This factor can be ignored.

  When selecting node specifications, you also need to consider the heap memory overhead of each node. Heap memory allocation policy: The size of the heap memory of each node is half of the node's physical memory, and the maximum size is 31 GB.

  Example:

  When creating a graph index based on the SIFT10M dataset (128-dimensional vector, 10 million records), if neighbors is set to 32, the off-heap memory required is as follows: mem_size = (128 x 4 + 32 x 4) x 10,000,000 + delta ≈ 6 GB. If one index replica is needed, at least 12 GB (6 x 2) of off-heap memory is required. In this case, select one 8U32G or two 8U16G node for the cluster.

Creating a Cluster

The procedure for creating a vector cluster is the same as that for creating any other regular search cluster. For details, see Creating an Elasticsearch Cluster (New Version).

Pay attention to the following key parameters:

• Cluster Version: Select 7.6.2 or 7.10.2. Only clusters of these two versions support the built-in CSS vector search engine.
• Node Specifications: Select the node specifications based on the result of cluster memory planning.

(Optional) Configuring the Circuit Breaker

The off-heap memory circuit breaker is enabled by default. You can enable or disable it and adjust its threshold based on service requirements. The command is as follows:

PUT _cluster/settings
{
  "persistent": {
    "native.cache.circuit_breaker.enabled": "true",
    "native.cache.circuit_breaker.cpu.limit": "80%"
  }
}