Vector DB on IC exploration

[komal-sai-yral]

No description provided.

[komal-sai-yral]

hnsw-rs

• mmap-rs not wasm compatible
  • on removal, other issue of cpu_time::ProcessTime; (not essential for vector computation)

Oasys-db

• wasm error - simsimd (essential for vector computation)

Elna DB

• build_index is failing after insertion of 100 records in local

usearch

• wasm32 error

Things to try:

• arcmind (not clear on which algo it uses)
• vectune (prev attempt to deploy canister not success)
• hnsw-rs remove cpu_time (cpu_time spread across the codebase)

[komal-sai-yral]

created issues in the crate hnsw-rs and oasysdb github. Will be actively tracking usearch WASM support PR - unum-cloud/usearch#331

ic-vectune seems confusing. Vectune on the other hand has to be built after every addition and for 200,000 takes ~20 secs

[Natasha-GB]

• Rely on third party for now and monitor issues, if any
• 1 more day on vectune - check incremental indexing
• Unblock Jay's task

[komal-sai-yral]

• incremental indexing works
• each insert call takes ~1-2 secs in local

Issues:

• after roughly 180 vectors in local
  • Exception: Rejected: Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister trapped: unknown
• centroid
  • build_index in local for just 1 vector
    • Canister exceeded the limit of 40000000000 instructions for single message execution.

[harshita-srivastava-yral]

• Scrap the idea of exploring more on Vectune.
  Next Steps:

1. Discord communication to become first tester for commercial use
2. Set notification for pending pull request

[komal-sai-yral]

usearch - unum-cloud/usearch#328
discord convo - https://discord.com/channels/1063947616615923875/1265163307858264236/1265535292794998857

oasys - edwinkys/oasysdb#106 (they fixed this but another dependency still has compatibity issues)

[komal-sai-yral]

Next steps - Check with oasysdb folks about plans to support wasm32