experiment: run DSPy without Deno - #77
Draft
isaacbmiller wants to merge 3 commits into
Draft
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-019fb0dc-8d11-7510-abc9-4bc180e4e6e4 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
Amp-Thread-ID: https://ampcode.com/threads/T-019fb0dc-8d11-7510-abc9-4bc180e4e6e4 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
Amp-Thread-ID: https://ampcode.com/threads/T-019fb0dc-8d11-7510-abc9-4bc180e4e6e4 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Measure the CI and compatibility impact of removing Deno/Pyodide entirely and using Monty as DSPy's only first-party code interpreter.
This is an experiment, not a merge proposal. The base is pinned to Stanford DSPy main at
c69136b2for comparison with both upstream CI and the Monty-default/Deno-compatible experiment in #76.Changes
MontyInterpreterby default in RLM, ProgramOfThought, and CodeActPythonInterpreter, its Deno runner, public export, backend-specific tests, and docsNet diff: 111 insertions, 2,516 deletions.
Validation
171 passed, 2 skipped1164 passed, 247 skipped, 2 xfailedin 40.23s35 passedin 7.52sgit diff --check: passCI measurement
Comparisons:
Across both Monty-only runs, the median test job is 122s: 79s (39%) below the Deno baseline and 73s (37%) below the Monty-default/retained-Deno experiment.
Step timing explains the gain. The normal suite remains approximately flat (median 92s baseline vs 96s Monty-only on the second run), while the optional/Deno phase falls from a baseline median of 91s to 13s after the 105 Deno tests are removed. The speedup is therefore specifically the eliminated Deno compatibility suite, not faster unrelated DSPy tests.
*Each run had one unrelated ~120s normal-suite outlier on a different Python version. Those outliers make total workflow wall time misleading (258s and 246s respectively), but do not affect the repeatable per-job median or the 78s reduction in the former extra+Deno phase.