Problem or use case
Sometimes the user wants to dig up the exact lines of a document for accurate reference (e.g. mathematical lemmas, legal/financial details), and it would be useful to have a converter that cache documents in the most legible formats. Converting to HTML yields around 15-40% savings, while MarkDown would yield about 35-55%, keeping all the LaTeX equations + graphs/figures.
Proposed solution
Something based on this line of logic but more local pdf2htmlEX/pdf2htmlEX#209
Docling/MinerU/Markitdown/Marker/Unstructured all look interesting as repos as well but they are more general
Area
Memory / knowledge graph
Alternatives considered
Doing them manually seems to be a hassle when it can be automated into a pipeline (get PDF, convert to HTML/MD and cache that for later, agent will re-read text if KG or VectorDB deviates)
Problem or use case
Sometimes the user wants to dig up the exact lines of a document for accurate reference (e.g. mathematical lemmas, legal/financial details), and it would be useful to have a converter that cache documents in the most legible formats. Converting to HTML yields around 15-40% savings, while MarkDown would yield about 35-55%, keeping all the LaTeX equations + graphs/figures.
Proposed solution
Something based on this line of logic but more local pdf2htmlEX/pdf2htmlEX#209
Docling/MinerU/Markitdown/Marker/Unstructured all look interesting as repos as well but they are more general
Area
Memory / knowledge graph
Alternatives considered
Doing them manually seems to be a hassle when it can be automated into a pipeline (get PDF, convert to HTML/MD and cache that for later, agent will re-read text if KG or VectorDB deviates)