LLM PDF Parsing + LLM Guided Retrieval#255

Draft

KastanDay wants to merge 45 commits intomainfrom llm-pdf-parsing

+3,995-1

Commits on Apr 18, 2024

Commits on Apr 19, 2024

Commits on Apr 21, 2024

Use AllenAI's Grobid wrapper, parse results into pristine database entries. TODO: upload to SQLite
KastanDay
committed

Commits on Apr 22, 2024

Working Grobid PDF parsing and upload to SQLite, all unique section names get own columns
KastanDay
committed
Fast, parallel, Grobid parsing, all in Python. Need to decide how to handle diverse column names
KastanDay
committed

Commits on Jun 5, 2024

LLM Parsing
jackhaohuang
committed

Commits on Jun 7, 2024

implemented embedding
jackhaohuang
committed

Commits on Jun 11, 2024

implemented parallel
jackhaohuang
committed

Commits on Jun 12, 2024

cleaning sqlite functions
jackhaohuang
committed

Commits on Jun 13, 2024

Commits on Jun 15, 2024

clean up
jackhaohuang
committed

Commits on Jun 18, 2024

fixed ULID and Embedding
jackhaohuang
committed

Commits on Jun 22, 2024

started guided retrival
jackhaohuang
committed

Commits on Jul 2, 2024

Commits on Jul 8, 2024

Commits on Jul 11, 2024

change sections to outline, adding abstract, and minio name
jackhaohuang
committed

Commits on Jul 18, 2024

Commits on Aug 1, 2024

Update grobid for higher concurrency; add context_idx to sql tables
jackhaohuang
committed

Commits on Aug 3, 2024

Add posthog logging, silently allow missing title and author errors, add grobid speedtest
KastanDay
committed

Commits on Aug 28, 2024

Parallelism and error handling improvements
jackhaohuang
committed

Commits on Oct 11, 2024

Added Caddy reverse proxy; improved error logs; speed is up to 14.5 PDFs per second with 1 delta node not even working that hard
jackhaohuang
committed

Commits on Nov 11, 2024