Code

Various Scale Things#22

Code

Closed

mitchellgordon95 wants to merge 26 commits intolucidrains:mainlucidrains/RETRO-pytorch:mainfrom latitudegames:mainCopy head branch name to clipboard

+241-81

Commits

Commits on Mar 11, 2022

• Build the faiss index ourselves instead of using autofaiss

  mitchellgordon95

  committed
  d605483

• Train the faiss index using gpus

  mitchellgordon95

  committed
  e3ccd6c

• Add transformers to the pip dependencies

  mitchellgordon95

  committed
  52dd5d9

Commits on Mar 12, 2022

• If we're using GPUs, mask should also be sent to GPU

  mitchellgordon95

  committed
  8bcc873

• Add worker_id to chunks_to_embeddings to facilitate parallel processing

  mitchellgordon95

  committed
  e6cc9c4

• If the faiss index is already pre-computed, use it when calculating knns

  mitchellgordon95

  committed
  5bd529c

Commits on Apr 17, 2022

• Merge branch 'main' of github.com:lucidrains/RETRO-pytorch into main

  mitchellgordon95

  committed
  f1aaf6c

Commits on Apr 24, 2022

• Add support for jsonl parsing to chunk creation

  mitchellgordon95

  committed
  74f3048

• Merge remote-tracking branch 'upstream/main' into main

  mitchellgordon95

  committed
  ad42726

• Add some logging when processing chunks

  mitchellgordon95

  committed
  c7641b9

Commits on Apr 25, 2022

• Gracefully handle hitting max chunks / seqs during tokenization

  mitchellgordon95

  committed
  03e0b39

Commits on May 7, 2022

• Add BertEmbeds class

  mitchellgordon95

  committed
  b20f775

• Use BertEmbeds class instead of numpy.memmap

  mitchellgordon95

  committed
  a53a5df

• Merge branch 'main' of github.com:lucidrains/RETRO-pytorch into main

  mitchellgordon95

  committed
  586d4d0

• Add context management to BertEmbeds class

  mitchellgordon95

  committed
  424d790

Commits on May 18, 2022

• Don't randomly select embeds for training, just use the first N

  mitchellgordon95

  committed
  118aa50

Commits on Jun 5, 2022

• Add embeddings to the index in chunks, so we don't run out of RAM

  mitchellgordon95

  committed
  5124511

Commits on Jun 6, 2022

• Add a safe-guard index write

  mitchellgordon95

  committed
  1347dba

Commits on Jun 19, 2022

• Don't put the index in a tmp folder, make the path explicit

  mitchellgordon95

  committed
  1a5a92b

• Add pre-trained index option when indexing embeddings

  mitchellgordon95

  committed
  8dfddf0

• Remove autofaiss dependency

  mitchellgordon95

  committed
  ff4cca5

Commits on Jun 26, 2022

• Add option to set FAISS thread count

  mitchellgordon95

  committed
  6d106a1

Commits on Jun 28, 2022

• Add checkpointing when adding embeddings to index

  mitchellgordon95

  committed
  b1e36ac

Commits on Jun 29, 2022

• Don't use OPQ dimensionality reduction

  mitchellgordon95

  committed
  0ff8a12

• Do OPQ dimensionality reduction, but make sure we're targeting 16 subvectors

  mitchellgordon95

  committed
  214d074

Commits on Aug 3, 2022

• Update retrieval.py

  mitchellgordon95

  authored
  10be243