Code

Narek/external index storage#335

Code

Open

Ngalstyan4 wants to merge 80 commits intounum-cloud:main-devunum-cloud/usearch:main-devfrom Ngalstyan4:narek/external-index-storageNgalstyan4/usearch:narek/external-index-storageCopy head branch name to clipboard

+1,506-955

Commits

Commits on Jan 29, 2024

• Pull in simsimd headers only if simsimd feature is enabled

  Ngalstyan4

  committed
  d416092

• Fix simd, openmp anf fp16 option value propagation to the compiler

  Show description for 61db8e5
  Ngalstyan4

  committed
  61db8e5

• Fix AVX512 detection logic

  Show description for 211a103
  Ngalstyan4

  committed
  211a103

• Avoid including simsimd before index_plugins

  Show description for c4394ce
  Ngalstyan4

  committed
  c4394ce

• Fix bench_cpp binary name in benchmark documentation

  Ngalstyan4

  committed
  c71e9c6

Commits on Jan 30, 2024

• Update simsimd to fix fp16 type inference

  Ngalstyan4

  committed
  b10c582

• Update SimSIMD to v3.7.5

  Ngalstyan4

  committed
  1bd4842

• Add clang-tidy clarification. todo:: is this needed?

  Ngalstyan4

  committed
  daf7ca2

• Initial trials to move node allocation outside of index.hpp

  Ngalstyan4

  committed
  2df158c

• Move storage parameter to right before metric

  Ngalstyan4

  committed
  d4fb277

• Slowly moving index storage outside of index.hpp

  Ngalstyan4

  committed
  aac0233

• Make external storage more functional by adding a vector storage which is actually used by index.hpp

  Ngalstyan4

  committed
  8209b52

• Move storage to a separate class and revert stats back to index

  Ngalstyan4

  committed
  a16e11f

• External storage with usearch working

  Show description for 76325fd
  Ngalstyan4

  committed
  76325fd

• Add size to bitset_gt

  Ngalstyan4

  committed
  c36756b

• ammend to the one before the last one

  Ngalstyan4

  committed
  9b30845

• Remove per-function call storage argument in favor of global storage

  Ngalstyan4

  committed
  b78f4f9

• Get rid of global storage lock

  Ngalstyan4

  committed
  ec9369a

• Fix memory leaks

  Ngalstyan4

  committed
  7e966b5

• Rename capacity to size in added bitset_gt size

  Ngalstyan4

  committed
  41f6039

• Move node sizing functions to node_t definition

  Ngalstyan4

  committed
  c28751a

• add assert and get rid of strange resizing logic

  Ngalstyan4

  committed
  e2d2670

• Improve node sizing api

  Ngalstyan4

  committed
  d81d58d

• Improve node_t interface, move precompute_ inside

  Ngalstyan4

  committed
  f1b47f3

• Clean node allocation API

  Ngalstyan4

  committed
  b48a657

• Make storage pass-by reference for ergonimics

  Ngalstyan4

  committed
  ee9bb58

• Add exchange fix an Q for Ashot

  Ngalstyan4

  committed
  73cf2a6

• get rid of underscores in storage function names

  Ngalstyan4

  committed
  731dfa2

• Get rid of tape_allocator from index and improve dummy storage

  Ngalstyan4

  committed
  a130ae8

• Add wip storage_v2

  Ngalstyan4

  committed
  204535c

• Move vector storage to storage_ class

  Ngalstyan4

  committed
  5fdee31

• Move vector storage to storage_v2 and fix compile warnings

  Show description for 6dab7ab
  Ngalstyan4

  committed
  6dab7ab

• Move vectors to external storage, keep similar interface

  Ngalstyan4

  committed
  3c11340

• Move storage providers to separate files

  Ngalstyan4

  committed
  d303be1

• Finish storage interface for loading from file

  Ngalstyan4

  committed
  0dcf791

• Fix dynamic allocation issue in storage

  Ngalstyan4

  committed
  854a11c

• Clean up Existing diff without adding any new features

  Ngalstyan4

  committed
  28c9a04

• Fix: Make sure index to stream saving callbacks take const arguments

  Ngalstyan4

  committed
  098f875

• More high level cleanup

  Ngalstyan4

  committed
  84734d6

• Move index load to under storage API

  Ngalstyan4

  committed
  75a3cc4

• Add view() support from storage

  Ngalstyan4

  committed
  b5d0a50

• Get rid of the duplicate precompute_ and use the one from node_t everywhere

  Ngalstyan4

  committed
  e36bd1c

• Get rid of global node_head_bytes and use the equivalent node_t::head_size_bytes() everywhere

  Ngalstyan4

  committed
  754d9b7

• Remove unnecessary code changes (comments and formatting)

  Ngalstyan4

  committed
  02ecba4

• Remove compaction API for initial storage PR

  Ngalstyan4

  committed
  f26cde9

• Remove more useless changes

  Ngalstyan4

  committed
  846355c

• Remove unused header

  Ngalstyan4

  committed
  c42c2f7

• Remove useless changes

  Ngalstyan4

  committed
  f969d4e

• Attempt adding storage provider typechecking

  Ngalstyan4

  committed
  4af55fd

• Add const-ness and noexcept enforcement to HAS_FUNCTION macro

  Ngalstyan4

  committed
  1448096

• Add helper macros for various signature assertions

  Ngalstyan4

  committed
  b13269e

• Add more functions for storage API enforcement

  Ngalstyan4

  committed
  7717c29

• Get rid of the old approach for API enforcement

  Ngalstyan4

  committed
  64996a7

• remove last remnant of class typechecking with the old approach

  Ngalstyan4

  committed
  36f7109

• Add Storage type enforcement API comments

  Ngalstyan4

  committed
  21f6b76

• Add comments and add the rest of Storage interface enforcement

  Ngalstyan4

  committed
  33220d5

• Move viewed_file_ state to storage_

  Ngalstyan4

  committed
  f37bb7a

• Fix first bug wound by typechecker

  Ngalstyan4

  committed
  4291071

• Fix nodes_ allocator to be what in original storage_v2 it was

  Ngalstyan4

  committed
  08db9ec

• Add a simple storage interface that uses std containers

  Ngalstyan4

  committed
  338e2fe

• Add storage choice to the tests

  Ngalstyan4

  committed
  c6ad76d

• Cleanup and rename std storage

  Ngalstyan4

  committed
  25c7ca4

• Improve std storage code

  Ngalstyan4

  committed
  6ff2c8f

• Add storage argument to index_dense_gt as well

  Ngalstyan4

  committed
  a7cc87c

• Add a note on later using node_t from storage_, instead of re-including it in index_* classes

  Ngalstyan4

  committed
  d74263a

• Add note on exchange not working with std::allocator template argument

  Ngalstyan4

  committed
  69cdbdb

• Add a todo on storage::memory_usage

  Ngalstyan4

  committed
  96d9c53

• Rename storage_v2 -> storage_v2_at

  Ngalstyan4

  committed
  e024fca

• Add a note on index_* not being movable

  Ngalstyan4

  committed
  9b979bd

• Rename view sub-interfaces

  Ngalstyan4

  committed
  6e11258

• Add setters to the enforced storage API

  Ngalstyan4

  committed
  ea5cdbb

• Fix std storage commentdoc

  Ngalstyan4

  committed
  a745439

• Cleanup and improve storage naming in test.cpp

  Ngalstyan4

  committed
  de89cef

• Bugfix: make sure default storage types allocate memory with proper alignment

  Ngalstyan4

  committed
  72d68a0

• Add proper alignment for std_storage so UBsan will not complain from view()

  Ngalstyan4

  committed
  87ab073

• Bring back original clang-tidy

  Ngalstyan4

  committed
  b2a3d3a

• Update vectors allocator back to default since custom one has compile issues under some conditions

  Ngalstyan4

  committed
  c504e18

• Store storage pointer instead of storage reference in typed index to allow move

  Ngalstyan4

  committed
  8dccd68

• Add (quite questionable) move constructors

  Ngalstyan4

  committed
  cae7210

• Initialize skiped usearch opt opts.multi

  Ngalstyan4

  committed
  ec3ed82