Add CodSpeed performance benchmarks for core DSPy operations - #54
Add CodSpeed performance benchmarks for core DSPy operations#54codspeed-hq[bot] wants to merge 1 commit into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| - name: Install dependencies | ||
| run: uv sync --dev -p .venv --extra dev | ||
| - name: Run benchmarks | ||
| uses: CodSpeedHQ/action@b16b7f2241a8564d005126c814839e9e990045a0 # v4 |
| - name: Install dependencies | ||
| run: uv sync --dev -p .venv --extra dev | ||
| - name: Run benchmarks | ||
| uses: CodSpeedHQ/action@b16b7f2241a8564d005126c814839e9e990045a0 # v4 |
Greptile SummaryThis PR introduces continuous performance tracking for DSPy by adding 41
Confidence Score: 4/5Safe to merge; the workflow and dependency changes are solid, and the only substantive defect is confined to one benchmark function that under-measures what it claims to measure. The json roundtrip benchmark silently omits deserialization — any future performance analysis relying on that metric will see only half the cost and could lead to incorrect optimization conclusions. All other benchmarks, the CI workflow, and the dependency additions look correct and consistent with existing conventions. tests/benchmarks/test_benchmarks.py — specifically the roundtrip benchmark and the unused fixture. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[GitHub Event\npush / PR / workflow_dispatch] --> B[codspeed.yml workflow]
B --> C[Checkout + setup-python 3.12]
C --> D[Install uv + create .venv]
D --> E[uv sync --dev --extra dev]
E --> F[CodSpeedHQ/action v4\nmode: simulation]
F --> G[uv run pytest tests/benchmarks/ --codspeed]
G --> H1[TestExampleBenchmarks\n11 benchmarks]
G --> H2[TestSignatureBenchmarks\n11 benchmarks]
G --> H3[TestAdapterBenchmarks\n16 benchmarks]
G --> H4[TestSerializationBenchmarks\n3 benchmarks]
H1 & H2 & H3 & H4 --> I[CodSpeed\nPerformance Report + Flamegraphs]
Reviews (1): Last reviewed commit: "Add CodSpeed performance benchmarks and ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def test_example_json_roundtrip(self, benchmark, nested_example): | ||
| """Benchmark JSON serialization and deserialization of an Example.""" | ||
|
|
||
| def roundtrip(): | ||
| d = nested_example.toDict() | ||
| _ = json.dumps(d) | ||
|
|
||
| benchmark(roundtrip) |
There was a problem hiding this comment.
json.loads is never called — only toDict() + json.dumps are measured. Anyone relying on this benchmark to evaluate the full serialization/deserialization cycle will see only half the picture; the deserialization cost (which can be comparable) is silently omitted.
| def test_example_json_roundtrip(self, benchmark, nested_example): | |
| """Benchmark JSON serialization and deserialization of an Example.""" | |
| def roundtrip(): | |
| d = nested_example.toDict() | |
| _ = json.dumps(d) | |
| benchmark(roundtrip) | |
| def test_example_json_roundtrip(self, benchmark, nested_example): | |
| """Benchmark JSON serialization and deserialization of an Example.""" | |
| def roundtrip(): | |
| d = nested_example.toDict() | |
| s = json.dumps(d) | |
| _ = json.loads(s) | |
| benchmark(roundtrip) |
| @pytest.fixture | ||
| def chat_completion_text(qa_signature_class): | ||
| return "[[ ## answer ## ]]\nParis is the capital of France.\n\n[[ ## completed ## ]]" | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Example / Prediction benchmarks |
There was a problem hiding this comment.
chat_completion_text fixture is never referenced by any benchmark test, yet it declares qa_signature_class as a parameter — causing pytest to eagerly instantiate that fixture (a class with DSPy field metaclass machinery) on every test collection run, for no benefit. It should be removed.
| @pytest.fixture | |
| def chat_completion_text(qa_signature_class): | |
| return "[[ ## answer ## ]]\nParis is the capital of France.\n\n[[ ## completed ## ]]" | |
| # --------------------------------------------------------------------------- | |
| # Example / Prediction benchmarks | |
| # --------------------------------------------------------------------------- | |
| # Example / Prediction benchmarks |
1 participant
Summary
This PR sets up continuous performance tracking for DSPy using CodSpeed. It adds 41 benchmarks targeting the most performance-critical, CPU-bound code paths in the framework, along with a dedicated GitHub Actions workflow that runs on every push and pull request.
Changes
Benchmarks (
tests/benchmarks/test_benchmarks.py)41 benchmarks organized across four areas:
Example / Prediction -- Core data container operations that run on every DSPy call:
with_inputs,inputs/labelssplittingtoDictrecursive serialization, hash computation, dict-like iterationSignature -- Parsing and manipulation of DSPy signatures:
make_signaturefrom simple and complex typed strings (AST parsing)append,prepend,delete,with_instructionsdump_state/load_state), equality comparisoninfer_prefixfor camelCase and snake_case conversionAdapter formatting and parsing -- The ChatAdapter hot path that runs on every LM call:
format_field_description,format_field_structure,format_user_message_content,format_assistant_message_contentparsefor single-field and multi-field completionsformat_field_value(string and list),serialize_for_json,translate_field_typeparse_valuefor string, int, and list typesget_annotation_name,get_field_description_stringSerialization -- Deep copy and JSON roundtrip for nested structures
CI Workflow (
.github/workflows/codspeed.yml)main, pull requests, andworkflow_dispatch(for CodSpeed backtesting)Other
pytest-codspeedas a dev dependencyNext steps