float_vector.md

• Creation
  • HNSW options
  • IVF options
• Float vector fields in selection results
• KNN search
• Rank
• Query examples
• Environment variables affecting vector indexes
• Additional action commands
  • Rebuilding clusters for IVF index
  • Removing disk cache for ANN indexes

Creation

Reindexer supports three types of vector indexes: hnsw (based on HNSWlib), ivf (based on FAISS IVF FLAT) and brute force (vec_bf), and three types of metrics for calculating the measure of similarity of vectors: inner_product, l2 and cosine.

For all types of vector indexes, the metric and the dimension should be explicitly specified. Vectors of only the specified dimension can be inserted and searched in the index.

The initial size start_size can optionally be specified for brute force и hnsw indexes, which helps to avoid reallocations and reindexing. The optimal value is equal to the size of the fully filled index. A much larger start_size value will result in memory overuse, while a much smaller start_size value will slow down inserts. Minimum and default values ​​are 1000.

HNSW options

Hnsw index is also configured with the following parameters:

• m - https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md#construction-parameters. Optional, range of values is [2, 128], default value is 16.
• ef_construction - https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md#construction-parameters. Optional, range of values is [4, 1024], default value is 200.
• multithreading - 0 or 1. 0 - single thread insert mode is used (uses less memory and CPU, but inserting into the index is slower). 1 - multithread insertion mode is used for transactions, but single inserts are still single thread (the index itself takes up more memory, but has a high parallelism factor). The number of insertion threads is specified via tx_vec_insertion_threads in namespace #config, by default 4 threads.
  Optional, default value is 0.

IVF options

For ivf index, the number of vectors (called centroids) should be specified, that will be chosen to partition the entire set of vectors into clusters. Each vector belongs to the cluster of the centroid closest to itself. The higher the centroids_count, the fewer vectors each cluster contains, this will speed up the search but will slow down the index building.
Required, range of values is [1, 65536]. It is recommended to take values of the order between $4 * \sqrt{number: of: vectors: in: the: index}$ and $16 * \sqrt{number: of: vectors: in: the: index}$.

Examples

// For a vector field, the data type must be array or slice of `float32`.
type Item struct {
	Id      int           `reindex:"id,,pk"`
	// In case of a slice, `dimension` should be explicitly specified in the field tag.
	VecBF   []float32     `reindex:"vec_bf,vec_bf,start_size=1000,metric=l2,dimension=1000"`
	// In case of an array, `dimension` is taken to be equal to the size of the array.
	VecHnsw [2048]float32 `reindex:"vec_hnsw,hnsw,m=16,ef_construction=200,start_size=1000,metric=inner_product,multithreading=1"`
	VecIvf  [1024]float32 `reindex:"vec_ivf,ivf,centroids_count=80,metric=cosine"`
}

When adding a vector index to an existing namespace, the field on which the index will be built must be empty or contain an array of numbers of length equal to the dimension of the index.

ivfOpts := reindexer.FloatVectorIndexOpts {
	Metric: "l2",
	Dimension: 1024,
	CentroidsCount: 32,
}
indexDef := reindexer.IndexDef {
	Name: "vec",
	JSONPaths: []string{"vec"},
	IndexType: "ivf",
	FieldType: "float_vector",
	Config: ivfOpts,
}
err := DB.AddIndex("ns_name", indexDef)
if err != nil {
	panic(err)
}

Float vector fields in selection results

By default, float vector fields are excluded from the results of all queries to namespaces containing vector indexes. If you need to get float vector fields, you should specify this explicitly in the query. Either by listing the required fields, or by requesting the output of all vectors().

• By default, without explicitly requesting vector fields:

SELECT * FROM test_ns;

db.Query("test_ns")
db.Query("test_ns").Select("*")

Result:

{"id": 0}

• Requesting for specific fields:

SELECT *, VecBF FROM test_ns;

db.Query("test_ns").Select("*", "VecBF")

Result:

{"id": 0, "vec_bf": [0.0, 1.1, ...]}

• Requesting all vector fields:

SELECT *, vectors() FROM test_ns;

db.Query("test_ns").Select("*", "vectors()")
db.Query("test_ns").SelectAllFields()

Result:

{"id": 0, "vec_bf": [0.0, 1.1, ...], "vec_hnsw": [1.2, 3.5, ...], "vec_ivf": [5.1, 4.7, ...]}

KNN search

The only filtering operation of float vector fields is KNN. It is not possible to use multiple KNN filters in a query, and it is impossible to combine filtering by KNN and fulltext.

Parameters set for KNN-query depends on the specific index type. The only required parameter for KNN is k - the maximum number of documents returned from the index for subsequent filtering.

When searching by hnsw index, you can additionally specify the ef parameter. Increasing this parameter allows you to get a higher quality result (recall rate), but at the same time slows down the search. See description here. Optional, minimum and default values ​​are k.

When searching by ivf index, you can additionally specify the nprobe parameter - the number of clusters to be looked at during the search. Increasing this parameter allows you to get a higher quality result (recall rate), but at the same time slows down the search. Optional, should be greater than 0, default value is 1.

• SQL
  KNN search condition is written as a function KNN, which takes the vector field as its first argument, the desired vector as its second argument, and k, ef, and nprobe as named arguments at the end:

SELECT * FROM test_ns WHERE KNN(vec_bf, [2.4, 3.5, ...], k=100)
SELECT * FROM test_ns WHERE KNN(vec_hnsw, [2.4, 3.5, ...], k=100, ef=200)
SELECT * FROM test_ns WHERE KNN(vec_ivf, [2.4, 3.5, ...], k=100, nprobe=10)

• Go
  The knn query parameters are passed in the reindexer.BaseKnnSearchParam structure, or in specific structures for each index type: reindexer.IndexBFSearchParam, reindexer.IndexHnswSearchParam or reindexer.IndexIvfSearchParam. There are factory methods for constructing them:

func NewBaseKnnSearchParam(k int) (*BaseKnnSearchParam, error)
func NewIndexBFSearchParam(baseParam *BaseKnnSearchParam) (*IndexBFSearchParam, error)
func NewIndexHnswSearchParam(ef int, baseParam *BaseKnnSearchParam) (*IndexHnswSearchParam, error)
func NewIndexIvfSearchParam(nprobe int, baseParam *BaseKnnSearchParam) (*IndexIvfSearchParam, error)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(4291)
if err != nil {
	panic(err)
}
// brute force
db.Query("test_ns").WhereKnn("vec_bf", []float32{2.4, 3.5, ...}, knnBaseSearchParams)
// hnsw
hnswSearchParams, err := reindexer.NewIndexHnswSearchParam(100000, knnBaseSearchParams)
if err != nil {
	panic(err)
}
db.Query("test_ns").WhereKnn("vec_hnsw", []float32{2.4, 3.5, ...}, hnswSearchParams)
// ivf
ivfSearchParams, err := reindexer.NewIndexIvfSearchParam(10, knnBaseSearchParams)
if err != nil {
	panic(err)
}
db.Query("test_ns").WhereKnn("vec_ivf", []float32{2.4, 3.5, ...}, ivfSearchParams)

Rank

By default, the results of queries with KNN are sorted by rank, that is equal to requested metric values. For indexes with the l2 metric, from a lower value to a higher value, and with the inner_product and cosine metrics, from a higher value to a lower value. This is consistent with the best match for the specified metrics.

When it is necessary to get the rank-value of each document in the query result, it must be requested explicitly via the RANK() function in SQL or WithRank() in GO:

SELECT *, RANK() FROM test_ns WHERE KNN(vec_bf, [2.4, 3.5, ...], k=200)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(200)
if err != nil {
	panic(err)
}
db.Query("test_ns").WithRank().WhereKnn("vec_bf", []float32{2.4, 3.5, ...}, knnBaseSearchParams)

Result:

{"id": 0, "rank()": 1.245}

rank can also be used to sort by expression. Described in detail here.

Query examples

• Simple KNN query on hnsw index:

SELECT * FROM test_ns WHERE KNN(vec_hnsw, [2.4, 3.5, ...], k=100, ef=150)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(100)
if err != nil {
	panic(err)
}
hnswSearchParams, err := reindexer.NewIndexHnswSearchParam(150, knnBaseSearchParams)
if err != nil {
	panic(err)
}
db.Query("test_ns").WhereKnn("vec_hnsw", []float32{2.4, 3.5, ...}, hnswSearchParams)

• Simple KNN query on brute force index with rank requesting:

SELECT *, RANK() FROM test_ns WHERE KNN(vec_bf, [2.4, 3.5, ...], k=100)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(100)
if err != nil {
	panic(err)
}
db.Query("test_ns").
	WithRank().
	WhereKnn("vec_bf", []float32{2.4, 3.5, ...}, knnBaseSearchParams)

• Simple KNN query on ivf index with requesting of vector fields:

SELECT *, vectors() FROM test_ns WHERE KNN(vec_ivf, [2.4, 3.5, ...], k=200, nprobe=32)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(200)
if err != nil {
	panic(err)
}
ivfSearchParams, err := reindexer.NewIndexIvfSearchParam(32, knnBaseSearchParams)
if err != nil {
	panic(err)
}
db.Query("test_ns").
	SelectAllFields().
	WhereKnn("vec_ivf", []float32{2.4, 3.5, ...}, ivfSearchParams)

• id > 5 AND KNN по ivf индексу with requesting one vector field:

SELECT *, VecBF FROM test_ns WHERE id > 5 AND KNN(vec_ivf, [2.4, 3.5, ...], k=300, nprobe=32)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(300)
if err != nil {
	panic(err)
}
ivfSearchParams, err := reindexer.NewIndexIvfSearchParam(32, knnBaseSearchParams)
if err != nil {
	panic(err)
}
db.Query("test_ns").
	Select("*", "VecBF").
	Where("id", reindexer.GT, 5).
	WhereKnn("vec_ivf", []float32{2.4, 3.5, ...}, ivfSearchParams)

• Query with parentheses and ordering by expression with rank:

SELECT * FROM test_ns
WHERE
	id < 1000
	AND (
		id > 5
		AND KNN(vec_ivf, [2.4, 3.5, ...], k=10000, nprobe=32)
	)
ORDER BY 'rank() * 10 - id' DESC

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(10000)
if err != nil {
	panic(err)
}
ivfSearchParams, err := reindexer.NewIndexIvfSearchParam(32, knnBaseSearchParams)
if err != nil {
	panic(err)
}
db.Query("test_ns").
	Where("id", reindexer.LT, 1000).
	OpenBracket().
		Where("id", reindexer.GT, 5).
		WhereKnn("vec_ivf", []float32{2.4, 3.5, ...}, ivfSearchParams).
	CloseBracket().
	Sort("rank() * 10 - id", true)

Environment variables affecting vector indexes

• RX_IVF_MT - 0 or 1. If it is 1, then for IVF indexes a multithread search mode is used. Default value is 0.
• RX_IVF_OMP_THREADS - number of OpenMP threads, used during constructing clusters for IVF indices. Default value is 8.
• RX_DISABLE_ANN_CACHE - if set, reindexer turns off disk ANN-cache to build vector indexes when loading data from disk during startup (it is not recommended to turn off the cache, as it slows down startup).
• RX_CUSTOM_BLAS_LIB_NAME - allows to set a custom name for BLAS library, used in IVF indexes (for cases when reindexer cannot find it on its own).
• RX_CUSTOM_LAPACK_LIB_NAME - allows to set a custom name for LAPACK library, used in IVF indexes (for cases when reindexer cannot find it on its own).
• RX_TARGET_INSTRUCTIONS - expects one of the values avx512, avx2, avx or sse. Allows you to select a set of vector instructions used for distance calculation functions. By default, if it is not set, reindexer uses the "best" available for the CPU it is running on, but occasionally avx512 may be less effective than avx2/avx in multithread environment.

Additional action commands

These commands can be used by inserting them via upsert into the #config namespace.

Rebuilding clusters for IVF index

{"type":"action","action":{"command":"rebuild_ivf_index", "namespace":"*", "index":"*", "data_part": 0.5}}

The command can be useful for cases where the composition of vectors in the index has changed significantly and the current centroids do not provide sufficiently high-quality output.

• namespace - the target namespace (* - applies the command to all namespaces);
• index - the target index (* - applies the command to all suitable indexes in namespace);
• data_part - the portion of the total data in the index that will be used to build new clusters (range [0.0, 1.0]). The more data is used, the more accurate the resulting clusters will be, but the computation time for the centroids will also be longer.

Removing disk cache for ANN indexes

{"type":"action","action":{"command":"drop_ann_storage_cache", "namespace":"*", "index":"*"}}

The command can be useful for cases when you need to force the re-creation of the disk cache for ANN indexes, or disable it completely (using it together with the RX_DISABLE_ANN_CACHE environment variable).

• namespace - the target namespace (* - applies the command to all namespaces);
• index - the target index (* - applies the command to all suitable indexes in namespace).