September 2024 release

[whitead]

This comment is a draft for release notes below (this will be a merge commit)

New Features

• Automatic population of metadata: PDF metadata is automatically retrieved from a variety of providers, including adding bibtex, citation counts, journal quality assessments, and noting retractions
• full-text search: A major difference between our published work and this repo is ability to search over all of scientific literature. We've brought the OSS version closer by adding full-text keyword search via tantivy. Now you can index and search many papers before embdding, making it feasible to ingest many papers.
• unified settings management: You can now save/load settings and that makes it easier for us to distribute settings reflecting various tasks with PaperQA2. Examples are writing wikipedia articles, identifying contradictions, and obtaining structured data
• CLI: We've made a CLI that uses persistent parsings/indexes and makes it much easier to just ask questions of a folder of PDFs
• Litellm: We've adopted litellm as the LLM wrapper of choice. This means we now support many LLM APIs directly with only the model string changing. It also means we have "routers" now that can do fallbacks, api rate limiting, and retries.

Improvements

• More modern agent frameworks
• Reduction in dependencies
• Removed code duplicated by litellm
• Many improvements on code style and best practices

Regressions/Deprecation

We've removed the following features to keep our library focused:

• doc_match - we do not have enough data to support that this method actually helps for very large corpuses
• LangchainVectorStore - We no longer support more complex vector stores via Langchain like FAISS. Instead, we only support Numpy vector stores. We never found the paradigm of very large vector stores to be better than keyword search -> vector search -> LLM reranking and thus removed the code

Detailed Changes:

• typo by @oganm in typo #303
• Updated readme and models by @mskarlin in Updated readme and models #305
• Add Client (external API) Module For Enhanced Metadata by @mskarlin in Add Client (external API) Module For Enhanced Metadata #306
• Agentic workflows, locally indexed search, and CLI by @mskarlin in Agentic workflows, locally indexed search, and CLI #309
• Add new unpaywall provider by @mskarlin in Add new unpaywall provider #310
• Rollback search fields to list and dynamically compute md5 hash in tests by @mskarlin in Rollback search fields to list and dynamically compute md5 hash in tests #311
• Refactor to breakout config from rest of code by @whitead in Refactor to breakout config from rest of code #289
• Changed to rely on litellm for computing cost by @whitead in Changed to rely on litellm for computing cost #321
• Fixing LLMModel.axyz_iter type hints by @jamesbraza in Fixing LLMModel.axyz_iter type hints #324
• CLI Fixes by @whitead in CLI Fixes #322
• blackened code to prevent IDE scrolling by @jamesbraza in blackened code to prevent IDE scrolling #330
• Optimized import paths by @jamesbraza in Optimized import paths #331
• Removed pytest-mock plugin by @jamesbraza in Removed pytest-mock plugin #328
• Adding pytest-xdist plugin by @jamesbraza in Adding pytest-xdist plugin #329
• Passing mypy by @jamesbraza in Passing mypy #332
• Removing make_chain in favor of run_prompt by @jamesbraza in Removing make_chain in favor of run_prompt #325
• Readme updates by @mskarlin in Readme updates  #323
• Adding refurb tool, and lint CI by @jamesbraza in Adding refurb tool, and lint CI #333
• Fixing arg ordering after Removing make_chain in favor of run_prompt #325 by @jamesbraza in Fixing arg ordering after #325 #334
• Fixing parse_text after Passing mypy #332 by @jamesbraza in Fixing parse_text after #332 #335
• Fixing union attr error by @jamesbraza in Fixing union attr error #338
• Check if a journal name starts with the by @geemi725 in Check if a journal name starts with the  #320
• Fixing two more tests by @jamesbraza in Fixing two more tests #340
• All Ruff ANN autofixes by @jamesbraza in All Ruff ANN autofixes #341
• Adding in .mailmap by @jamesbraza in Adding in .mailmap #342
• Remove cassettes which aren't needed by @mskarlin in Remove cassettes which aren't needed  #339
• Add configs for contracrow + wikicrow by @mskarlin in Add configs for contracrow + wikicrow #336
• Removed LangchainVectorStore, llms extra, and fixing up README by @jamesbraza in Removed LangchainVectorStore, llms extra, and fixing up README #343
• Dropping requests dependency by @jamesbraza in Dropping requests dependency #346
• Removed html2text requirement by @jamesbraza in Removed html2text requirement #347
• Requiring Python 3.11+ by @jamesbraza in Requiring Python 3.11+ #348
• Did one revision at README by @whitead in Did one revision at README #344
• Renaming fitz to pymupdf by @mskarlin in Renaming fitz to pymupdf #350
• Better control flow in litellm_get_search_query by @jamesbraza in Better control flow in litellm_get_search_query #351
• Recurse into directories; catch empty documents by @sidnarayanan in Recurse into directories; catch empty documents #352
• Move configure_cli_logging such that it's not called twice by @mskarlin in Move configure_cli_logging such that it's not called twice #353
• Cleaning up dependencies by @jamesbraza in Cleaning up dependencies #354
• Fixed code in README by @whitead in Fixed code in README #355
• Added citation and paper URL by @whitead in Added citation and paper URL #357
• aviary and ldp for agents over langchain by @jamesbraza in aviary and ldp for agents over langchain #358
• Adds retraction status by @geemi725 in Adds retraction status #314
• Adding pylint by @jamesbraza in Adding pylint  #349
• Added account for cost info by @whitead in Added account for cost info #360

New Contributors

• @oganm made their first contribution in typo #303
• @geemi725 made their first contribution in Check if a journal name starts with the  #320
• @sidnarayanan made their first contribution in Recurse into directories; catch empty documents #352

Full Changelog: v4.9.0...vnew