Skip to content

A semantic search indexing system designed to efficiently retrieve top matching results from a database of 20 million documents. Given the embedding of a search query, it quickly identifies and returns the most relevant documents

License

Notifications You must be signed in to change notification settings

ZiadSheriif/IntelliQuery

Repository files navigation

IntelliQuery

📝 Table of Contents

📙 Overview

Given the embedding of the search query we can efficent get the top matching k results form DB with 20M document.The objective of this project is to design and implement an indexing system for a semantic search database.

Get Started

Infernce Mode

Check Final Notebook

https://github.com/ZiadSheriif/IntelliQuery/blob/main/Evaluate_ADB_Project.ipynb

Run Locally

Clone Repo

git clone https://github.com/ZiadSheriif/IntelliQuery.git

Install dependencies

pip install -r requirements.txt

Run Indexer

$ python ./src/evaluation.py

Methods

Inverted File Inedex (IVF)

This is out final Approach with Some Enhancements

  1. Changed MiniBatchKMeans to regular KMeans
  2. We calculate initial centroids with just the first chunk of data
  3. Introduced parallel processing for different regions

Local Sensitive Hashing (LSH)

Product Qunatization (PQ)

PQ-LSH

It Combines both LSH & PQ

🕴 Contributors


Ziad Sherif


Zeyad Tarek


Abdalhameed Emad


Basma Elhoseny

📃 License

This software is licensed under MIT License, See License for more information ©Ziad Sherif.

About

A semantic search indexing system designed to efficiently retrieve top matching results from a database of 20 million documents. Given the embedding of a search query, it quickly identifies and returns the most relevant documents

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •