Skip to content

Merge Cache and History, Reduce Critical Section - #32

Merged
krypticmouse merged 3 commits into
mainfrom
critical-section-reduction
Oct 15, 2025
Merged

Merge Cache and History, Reduce Critical Section#32
krypticmouse merged 3 commits into
mainfrom
critical-section-reduction

Conversation

@krypticmouse

Copy link
Copy Markdown
Owner

This pull request refactors the handling of language model (LM) instances and caching throughout the codebase, replacing the use of Arc<Mutex<LM>> with Arc<LM> for most operations. It also updates the LM construction and caching logic to be more asynchronous and robust, and modifies example usage to match these changes. These improvements simplify concurrency management, make the LM API easier to use, and ensure more consistent caching and history inspection.

LM and Adapter API Refactoring

  • Replaced Arc<Mutex<LM>> with Arc<LM> in the Adapter trait and its implementations, including ChatAdapter, removing unnecessary locking and simplifying usage. (crates/dspy-rs/src/adapter/mod.rs, crates/dspy-rs/src/adapter/chat.rs) [1] [2] [3] [4]
  • Updated global settings to store Arc<LM> instead of Arc<Mutex<LM>>, reflecting the new LM API. (crates/dspy-rs/src/core/settings.rs)

LM Construction and Caching

  • Refactored the LM builder to be asynchronous (build().await) and to initialize the cache handler as an Arc<Mutex<ResponseCache>> only when caching is enabled. Also, updated LM response and history tracking to use the cache rather than a local history vector. (crates/dspy-rs/src/core/lm/mod.rs) [1] [2] [3]
  • Updated DummyLM to match the new caching and history inspection APIs, including asynchronous cache insertion to avoid deadlocks. (crates/dspy-rs/src/core/lm/mod.rs)

Example Usage Updates

  • Updated all example files to use .build().await for LM construction, and removed unnecessary Mutex usage when passing LMs to configure. (crates/dspy-rs/examples/01-simple.rs, 03-evaluate-hotpotqa.rs, 04-optimize-hotpotqa.rs, 05-heterogenous-examples.rs, 06-oai-compatible-models-batch.rs, 07-inspect-history.rs, 08-optimize-mipro.rs, 09-gepa-sentiment.rs, 10-gepa-llm-judge.rs) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]
  • Updated history inspection in examples to use the new async cache-based API. (crates/dspy-rs/examples/07-inspect-history.rs, 10-gepa-llm-judge.rs) [1] [2]

Miscellaneous Improvements

  • Added a new push_message method to Chat for more flexible message handling. (crates/dspy-rs/src/core/lm/chat.rs)
  • Made minor improvements to imports and code clarity across affected files. [1] [2]

Fixes #31

@krypticmouse
krypticmouse merged commit 47ca636 into main Oct 15, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minimize Critical Section in LM call

1 participant