向量索引GsIVFFLAT
GsIVFFLAT索引是针对小数据量(万~百万级别)下的向量检索,支持在已创建含向量数据的列上创建单列索引,以及支持删除索引、支持数据变更后的索引自动更新。
前置条件
- 数据库的GUC参数enable_vectordb设置为on。
- 使用GsIVFFLAT索引时,数据不宜少于1*10^4条或超过2*10^6条,数据量过少时建议不建立索引,数据量过多时建议使用其他向量索引。
使用建议
- N条数据的表建议取ivf_nlist=4*sqrt(N),如果数据量较大需设置ivf_nlist2,则建议ivf_nlist2=sqrt(ivf_nlist)。
- GUC参数gsivfflat_probes建议设置为索引超参ivf_nlist的10%,根据所需查询精度和查询时延适当增加或降低gsivfflat_probes,查询时间与gsivfflat_probes/ivf_nlist的比值线性相关。同样,如创建索引时设置了ivf_nlist2,则GUC参数gsivfflat_secondary_probes建议设置为索引超参ivf_nlist2的10%,根据所需查询精度和查询时延适当增加或降低gsivfflat_secondary_probes,查询时间与gsivfflat_secondary_probes/ivf_nlist2的比值线性相关。
- 建议将所有需要查询的数据提前插入表中,最后再建立索引;如果决定先建立索引再导入数据,在数据导入完成后需要重建索引。
- 建立索引后如果数据变化(含插入、删除、更新)超过20%会产生数据分布漂移,导致查询精度下降与查询时间不稳定,建议每更新10%~20%数据重建一次索引。
- 建立索引时数据条目数若低于nlist取值,会将nlist视为1创建索引,并弹出提示。此时查询与全表扫描无异,包括准确率与时延。为使索引达到预期性能,需在数据导入完成后重建索引。需要注意,对表进行truncate操作后,GsIVFFLAT索引也会弹出相同的提示,使索引处于上述状态。如希望使索引重新可用,需要在数据重新导入后重建索引。
参数说明
| 参数名称 | 取值说明 | 参数描述 |
|---|---|---|
| ivf_nlist | 取值范围:1~65535 默认值:100 | 倒排列表的数量。 |
| ivf_nlist2 | 取值范围:0~65535 默认值:0 | 二级倒排列表数量。当ivf_nlist2取值为0或1时,不会创建二级倒排列表。 |
| num_parallels | 取值范围:1~64 默认值:1 | 构建索引并行计算数量,仅在NPU构建索引时生效。 |
在向量索引中一次性进行大量的删除或者更新操作,或者频繁的删除或者更新操作,可能会导致索引质量下降,索引膨胀,搜索时间变长的问题。
查询索引状态
使用函数gs_ivfflat_inspect('ivfflat_index_name')可查询当前GsIVFFLAT索引状态,包括索引磁盘使用、聚簇的数据分布等信息。每条信息的返回类型为record,需要用SELECT * FROM语句获取,函数参数为索引名,由建索引时指定或者按照默认规则生成,该名可以使用\d table_name或者从系统表pg_index进行查询。
对于包含向量索引的分区表,该函数会返回所有分区下索引的状态,查询某一具体分区的索引请使用gs_ivfflat_inspect('ivfflat_index_name', 'index_partition_name')只显示该分区索引的索引状态,索引分区名可以通过查询系统表pg_partition获取。
在使用查询索引状态函数分析索引数据过程时若有大量DML操作可能会造成分析结果有误差,执行过程中数据变化量若小于原数据总量的5%则输出结果可信度较高。
频繁调用gs_ivfflat_inspect函数可能会影响并发操作的性能。建议两次调用该函数的间隔时间不小于 10 秒,由此保证对并发操作性能的劣化影响控制在5%以内。
示例
--创建表 gaussdb=# CREATE TABLE "test" ("id" BIGINT PRIMARY KEY, "repr" floatvector (128)); --随机插入数据 gaussdb=# CREATE OR REPLACE FUNCTION float_random_array(dims int,range int) RETURNS float[] AS $$BEGIN RETURN ARRAY(SELECT (random() * range)::int FROM generate_series(1, dims));END;$$ LANGUAGE plpgsql; gaussdb=# INSERT INTO test SELECT i,floatvector(float_random_array(128,100)) FROM generate_series(1,10000) as i; --创建索引 gaussdb=# CREATE INDEX testivfflat ON test USING GSIVFFLAT(repr L2) WITH (IVF_NLIST = 256); --查询索引状态 gaussdb=# SELECT * FROM gs_ivfflat_inspect('testivfflat'); inspect_item | inspect_info -----------------------+-------------------------------------------------------------------- FixedPageBlk | MetaPage 0, ListGuard 1, BktGuard 2, BktScanGuard 3, ExtendGuard 4 TotalPageNum | 858 UsedPageNum | 858 Utilization Rate | 100.000000% FirstLevelListNum | 256 SecondLevelListNum | 0 FirstLvlTupCntMedian | 39 FirstLvlTupCntAverage | 39 FirstLvlTupCntStdDev | 24.3591 FirstLvlTupCntMax-K | 104,102,97,97,91 FirstLvlTupCntMin-K | 1,2,2,2,2 CanScanWithNPU | false (11 rows) --向量ANN检索 gaussdb=# set gsivfflat_probes = 25; gaussdb=# SELECT "id" FROM "test" ORDER BY "repr" <-> '[ 23.0,0.0,0.0,0.0,1.0,5.0,43.0,114.0,3.0,0.0,0.0,0.0,14.0,121.0,120.0,78.0,81.0,4.0,0.0,0.0,31.0,126.0,23.0,18.0,126.0,12.0,0.0,0.0,0.0,1.0,0.0,10.0,0.0,0.0,0.0,0.0,8.0,29.0,96.0,43.0,0.0,0.0,0.0,0.0,1.0,81.0,126.0,44.0,126.0,1.0,0.0,0.0,1.0,45.0,66.0,96.0,126.0,0.0,0.0,0.0,1.0,16.0,12.0,63.0,1.0,2.0,0.0,0.0,11.0,40.0,26.0,0.0,5.0,20.0,28.0,1.0,0.0,17.0,36.0,5.0,126.0,45.0,10.0,1.0,0.0,2.0,12.0,29.0,126.0,6.0,0.0,0.0,2.0,110.0,96.0,46.0,18.0,13.0,0.0,0.0,3.0,5.0,1.0,2.0,29.0,50.0,30.0,7.0,8.0,3.0,0.0,1.0,55.0,24.0,14.0,5.0,9.0,15.0,8.0,10.0,10.0,1.0,0.0,0.0,19.0,79.0,16.0,4.0 ]' LIMIT 10; --插入数据 gaussdb=# INSERT INTO test(id,repr) VALUES(99999999,'[ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 ]'); gaussdb=# SELECT COUNT(*) FROM test; --更新数据 gaussdb=# UPDATE test SET repr = '[ 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 ]' WHERE id = 99999999; gaussdb=# SELECT * FROM test WHERE id = 99999999; --删除数据 gaussdb=# DELETE FROM test WHERE id = 99999999; gaussdb=# SELECT COUNT(*) FROM test; --清理数据 gaussdb=# VACUUM test; --删除函数 gaussdb=# DROP FUNCTION float_random_array; --删除索引 gaussdb=# DROP INDEX testivfflat; --删除表 gaussdb=# DROP TABLE test;