Search across all Asymmetric Text Content Types in Parallel #232

debanjum · 2023-06-21T06:10:41Z

Allow searching across asymmetric text content types using threads
- Query time on my Mac averages 95ms latency (140ms at 90 percentile) across (Org, Markdown, Github, PDF and Music content types)
- This is not too much more than search for a single content type (maybe max ~50% latency increase?). Encoding query is what takes most of the time anyway and that's just done once like before, threading adds some overhead
- An average of 95 ms latency or 140ms at 90th percentile is inline with keeping an incremental search (search-as-you-type) experience
Put logic to remove filter terms from query in a defilter method for each filter
Encode query once during search to encode query once across all (asymmetric) content types
Search across all content types via the web and emacs interfaces in d5fb419 and 5c4eb95 respectively
Allow Khoj Chat to pull relevant data from across content types (without the perf hit). Khoj chat is only pulling data from a single content type currently

- Update API to return content from all enabled content types when type is not set to specific type in HTTP request param - To do this efficiently run the search queries in parallel threads

- Add new filter abstract method to remove filter terms from query - Use the filter method to remove filter terms, encode this defiltered query and pass it to the query methods of each search types TODO: Encoding query is still taking 100-200 ms unlike before. Need to investigate why

Use timer to measure time to encode queries and total search time

…thods

If no content-type selected in transient menu option, khoj.el queries khoj server without content-type parameter (t) set. This results in search across all enabled asymmetric search text content types

sabaimran

Sweet! This is a game changer 🎊 .

src/interface/emacs/khoj.el

src/khoj/configure.py

src/khoj/interface/web/index.html

src/khoj/search_filter/date_filter.py

src/khoj/search_type/text_search.py

…rch-across-all-asymmetric-text-content-types Conflicts: - src/khoj/routers/api.py: Use theirs

…server Allows moving logic to handle search across all content types to server from clients

- So when searching across content types (with content-type = "all") org-mode results get rendered differently than markdown, PDF etc. results - Set div class for each result separately instead of a single uber div for styling. This allows styling div of each result based on the content-type of that result - No need to create placeholder "all" content type on web interface as server is passing an all content type by itself

…n khoj.el - Add "all" as default content type when no content type retrieved from server

- Set image_search.query to async to use it with multi-threading This is same as text_search.query being set to an async method - Exit search early if no search_model is defined in state.model

The text, image search query method has become async. So async/await is required to get results correctly in tests etc

debanjum added 7 commits June 19, 2023 23:29

Search in parallel across all enabled content types requested via API

285d17a

- Update API to return content from all enabled content types when type is not set to specific type in HTTP request param - To do this efficiently run the search queries in parallel threads

Encode the asymmetric, symmetric search queries in parallel for speed

6d94d6e

Use timer to measure time to encode queries and total search time

Only search across content types that work with asymmetric search

0144e61

Pass default value matching argument types expected by text_search me…

1192e49

…thods

Use async/await to fix parallelization of search across content types

5c7c8d1

Update web interface to allow querying all content types at once

d5fb419

debanjum added the upgrade New feature or request label Jun 21, 2023

debanjum requested a review from sabaimran June 21, 2023 06:20

debanjum added 2 commits June 20, 2023 23:30

Improve null and type checks

2cd3e79

Search across all content types via khoj.el on Emacs

5c4eb95

If no content-type selected in transient menu option, khoj.el queries khoj server without content-type parameter (t) set. This results in search across all enabled asymmetric search text content types

debanjum force-pushed the parallelize-search-across-all-asymmetric-text-content-types branch from 9827221 to 5c4eb95 Compare June 21, 2023 07:10

debanjum force-pushed the master branch 3 times, most recently from 75ab8f2 to 6d4aad5 Compare June 21, 2023 08:57

debanjum linked an issue Jun 21, 2023 that may be closed by this pull request

Setup Unified Search to search across all content types #234

Closed

sabaimran approved these changes Jun 22, 2023

View reviewed changes

sabaimran reviewed Jun 22, 2023

View reviewed changes

src/khoj/search_type/text_search.py Outdated Show resolved Hide resolved

debanjum mentioned this pull request Jun 27, 2023

Support multiple Github repositories and support indexing of multiple file types #250

Merged

debanjum added 6 commits June 27, 2023 15:48

Null check config, log warning instead of error when configuring search

09f739b

Extract search request URL builder into js function in web interface

1b11d57

Use typing union in text_search for python 3.8 compatible type hinting

510bb7e

Merge branch 'master' of github.com:khoj-ai/khoj into parallelize-sea…

0636cea

…rch-across-all-asymmetric-text-content-types Conflicts: - src/khoj/routers/api.py: Use theirs

Create "all" search type for search across all content types on khoj …

212b1a9

…server Allows moving logic to handle search across all content types to server from clients

Fix createRequestUrl method signature to fetch results from khoj web

1773a78

debanjum force-pushed the parallelize-search-across-all-asymmetric-text-content-types branch from f62e4c2 to 5fc92e5 Compare June 29, 2023 03:28

debanjum added 4 commits June 28, 2023 22:07

Put each result under org heading when query for "all" content type i…

8eae7c8

…n khoj.el - Add "all" as default content type when no content type retrieved from server

Get any configured asymmetric search model to encode query for search

b1767f9

- Set image_search.query to async to use it with multi-threading This is same as text_search.query being set to an async method - Exit search early if no search_model is defined in state.model

Update client tests to expect "all" as a valid new content type

f516d12

debanjum added 2 commits June 28, 2023 22:07

Use async/await in tests for query method of text and image search

56ce97e

The text, image search query method has become async. So async/await is required to get results correctly in tests etc

Add trio, move freezegun, factory-boy to project test dependencies

5f7eaa7

debanjum force-pushed the parallelize-search-across-all-asymmetric-text-content-types branch from 5fc92e5 to f469d0a Compare June 29, 2023 05:08

Use logger.warning since logger.warn is deprecated

5f2717c

debanjum force-pushed the parallelize-search-across-all-asymmetric-text-content-types branch from f469d0a to 5f2717c Compare June 29, 2023 05:16

sabaimran approved these changes Jun 29, 2023

View reviewed changes

debanjum merged commit f272d45 into master Jun 29, 2023
10 checks passed

debanjum deleted the parallelize-search-across-all-asymmetric-text-content-types branch June 29, 2023 19:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Search across all Asymmetric Text Content Types in Parallel #232

Search across all Asymmetric Text Content Types in Parallel #232

debanjum commented Jun 21, 2023 •

edited

Loading

sabaimran left a comment

Search across all Asymmetric Text Content Types in Parallel #232

Search across all Asymmetric Text Content Types in Parallel #232

Conversation

debanjum commented Jun 21, 2023 • edited Loading

sabaimran left a comment

Choose a reason for hiding this comment

debanjum commented Jun 21, 2023 •

edited

Loading