Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the vector search performance with rows in the delta-layer #9600

Closed
Tracked by #9032
JaySon-Huang opened this issue Nov 11, 2024 · 1 comment
Closed
Tracked by #9032

Improve the vector search performance with rows in the delta-layer #9600

JaySon-Huang opened this issue Nov 11, 2024 · 1 comment
Assignees
@Lloyd-Pottiger
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Nov 11, 2024 •
edited
Loading

When there are updates to the table with vector index, the QPS of vector search will drop quickly. One of the reason is there is no vector index built on top of the delta layer. So it takes lots of disk IO and CPU to load and compute the distance.
Building vector index on the delta-layer could help improve the vector search performance with updates.
@JaySon-Huang JaySon-Huang mentioned this issue Nov 11, 2024
Closed
2 tasks
@JaySon-Huang JaySon-Huang added the type/enhancement The issue or PR belongs to an enhancement. label Nov 11, 2024
This was referenced Nov 11, 2024
Merged
Merged
Merged
Merged
@JaySon-Huang JaySon-Huang mentioned this issue Nov 11, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 13, 2024
@Lloyd-Pottiger
Storages: Optimize vector search in scenarios with updates (#9597)
9dc8374 
close #9599, ref #9600

Improve 75% the performance of vector search in scenarios with updates.

Signed-off-by: Lloyd-Pottiger <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue Nov 13, 2024
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 14, 2024
@ti-chi-bot @Lloyd-Pottiger @JaySon-Huang
Storages: Optimize vector search in scenarios with updates (#9597) (#…
932059d 
…9609)

close #9599, ref #9600

Improve 75% the performance of vector search in scenarios with updates.

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: Lloyd-Pottiger <[email protected]>
Co-authored-by: JaySon <[email protected]>
@Lloyd-Pottiger
Copy link
Contributor

Lloyd-Pottiger commented Nov 15, 2024 •
edited
Loading

Now the main bottleneck of vector search is MVCC, track in #9616

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lloyd-Pottiger Lloyd-Pottiger

Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JaySon-Huang @Lloyd-Pottiger