From 8c1fa13c7022db62cae18c3a60993e51c23bd067 Mon Sep 17 00:00:00 2001 From: "Corey J. Nolet" Date: Wed, 2 Oct 2024 18:11:57 -0400 Subject: [PATCH] Updates --- docs/source/tuning_guide.rst | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/docs/source/tuning_guide.rst b/docs/source/tuning_guide.rst index adba53958..7a0987a44 100644 --- a/docs/source/tuning_guide.rst +++ b/docs/source/tuning_guide.rst @@ -4,25 +4,17 @@ Tuning Guide ~~~~~~~~~~~~ -A Method for tuning and evaluating Vector Search Indexes At Scale in Locally Indexed Vector Databases +A Method for tuning and evaluating Vector Search Indexes At Scale in Locally Indexed Vector Databases. For more information on the differences between locally and globally indexed vector databases, please see :doc:`this guide `. The goal of this guide is to give users a scalable and effective approach for tuning a vector search index, no matter how large. Evaluation of a vector search index “model” that measures recall in proportion to build time so that it penalizes the recall when the build time is really high (should ultimately optimize for finding a lower build time and higher recall). -Objective -========= +For more information on the various different types of vector search indexes, please see our :doc:`guide to choosing vector search indexes ` -Give uswrs an approach for tuning a vector search index. Evaluation of a vector search index “model” that measures recall in proportion to build time so that it penalizes the recall when the build time is really high (should ultimately optimize for finding a lower build time and higher recall). +As much as 75% of users have told us they will not be able to tune a vector database beyond one or two simple knobs and we suggest that an ideal “knob” would be to balance training time and search time with search quality. The more time, the higher the quality, and the more needed to find an acceptable search performance. Even the 25% of users that want to tune are still asking for simple tools for doing so. These users also ask for some simple guidelines for setting tuning parameters, like :doc:`this guide `. -Output -====== -An example notebook which can be released in cuVS as an example for tuning an index, especially for CAGRA. +Since vector search indexes are more closely related to machine learning models than traditional databases indexes, one option for easing the parameter tuning burden is to use hyper-parameter optimization tools like `Ray Tune `_ and `Optuna `_. to verify this. -Background -========== +But how would this work when we have an index that's massively large- like 1TB? -Vector databases 101: Configuring Vector Search Indexes - -Many customers (Specifically AWS and Google) have told us that >75% of their users will not be able to tune a vector database beyond one or two simple knobs. They suggest that an ideal “knob” would be to balance training time with search quality. The more time, the higher the quality. For the <25% that wants to tune, they’ve asked for simple tools for tuning. They also ask for some simple guidelines for setting tuning parameters. -Strategy -Ray-tune and our Python APIs could be an option to verify this. We could write a notebook that takes some small subsampling from a dataset and does a parameter search on it. Then we actually evaluate random queries against the ground truth to test that the index params actually generalized well (I'm confident they will). +One benefit to locally indexed vector databases is that they tend to scale by breaking a larger set of vectors down into smaller small subsampling from a dataset and does a parameter search on it. Then we actually evaluate random queries against the ground truth to test that the index params actually generalized well (I'm confident they will). Getting Started with Optuna and RAPIDS for HPO — RAPIDS Deployment Documentation documentation