merge: release work from feat/feed-spec-schedule-rebalancing

.github/workflows/docs.yml

@@ -0,0 +1,50 @@
+name: Docs
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: "docs"
+  cancel-in-progress: true
+
+jobs:
+  build:
+    name: Build & Deploy Docs
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+
+      - name: Set up Python
+        run: uv python install 3.12
+
+      - name: Install dependencies
+        run: uv sync --dev --extra docs
+
+      - name: Build docs
+        run: uv run mkdocs build --strict
+
+      - name: Deploy to website repo
+        uses: cpina/github-action-push-to-another-repository@v1.7.2
+        env:
+          SSH_DEPLOY_KEY: ${{ secrets.DOCS_DEPLOY_KEY }}
+        with:
+          source-directory: site/
+          destination-github-username: ml4t
+          destination-repository-name: website
+          target-directory: static/docs/backtest/
+          target-branch: main
+          commit-message: "docs(backtest): update from ml4t/backtest@${{ github.sha }}"
+          user-name: ml4t-bot
+          user-email: bot@ml4trading.io

.gitignore

@@ -7,6 +7,7 @@ __pycache__/
 *.egg-info/
 dist/
 build/
+site/
 eggs/
 *.egg
 

CHANGELOG.md

@@ -0,0 +1,85 @@
+# Changelog
+
+## 0.1.0b11 - 2026-03-24
+
+### Added
+
+- `BacktestResult.predictions` and `BacktestResult.to_predictions_dataframe()` to preserve
+  the raw prediction or model-input DataFrame passed into the backtest.
+- `predictions.parquet` export/import support in `BacktestResult.to_parquet()` and
+  `BacktestResult.from_parquet()`.
+
+### Changed
+
+- Engine results now treat the raw `signals_df` input surface as predictions for downstream
+  diagnostics, matching `ml4t-diagnostic`'s current contract.
+- Parquet import falls back from legacy `signals.parquet` to the new predictions surface.
+- User guides and README now document the raw predictions surface for downstream analysis.
+
+### Validation
+
+- `uv run ruff check src/ml4t/backtest/result.py src/ml4t/backtest/engine.py tests/test_result.py tests/test_core.py`
+- `uv run pytest tests/test_result.py tests/test_core.py -q`
+- `uv run ty check`
+- `uv run python -m mkdocs build --strict`
+
+## 0.1.0b10 - 2026-03-24
+
+### Added
+
+- `BacktestConfig` support for serialized top-level `feed` and passthrough `metadata`
+  sections, enabling sparse input presets with generic provenance fields.
+- `BacktestResult.to_spec_dict()` for a resolved runtime snapshot containing the full
+  replayable config, library version, and realized run window.
+- `spec.yaml` export alongside `config.yaml` in `BacktestResult.to_parquet()`.
+
+### Changed
+
+- `BacktestConfig.to_dict()` now emits plain-data feed metadata that round-trips safely
+  through dict, YAML, and Parquet persistence workflows.
+- `BacktestResult.from_parquet()` now falls back to `spec.yaml` when `config.yaml` is absent.
+- User guides now document the config workflow for sparse input, resolved output, `feed`,
+  `metadata`, and reproducibility exports.
+
+### Validation
+
+- `uv run ruff check src/ml4t/backtest/config.py src/ml4t/backtest/result.py tests/test_broker.py tests/test_result.py`
+- `uv run pytest tests/test_broker.py tests/test_result.py -q`
+- `uv run ty check`
+- `uv run python -m mkdocs build --strict`
+
+## 0.1.0b9 - 2026-03-24
+
+### Added
+
+- Quote-aware `DataFeed` support for `price_col`, bid, ask, midpoint, and quote-size caches.
+- New `ExecutionPrice` sources: `price`, `bid`, `ask`, `quote_mid`, and `quote_side`.
+- Separate `mark_price` configuration for open-position marking.
+- `BacktestResult.to_fills_dataframe()` and persisted `fills.parquet` export/import.
+- `BacktestResult.to_portfolio_state_dataframe()` and persisted `portfolio_state.parquet`.
+- Quote context fields on `Fill` and summarized quote context on `Trade`.
+- Activity and exposure metrics: `num_fills`, `num_rebalance_events`, `unique_symbols_traded`,
+  `total_filled_notional`, `avg_turnover`, `max_turnover`, `avg_open_positions`,
+  and `max_open_positions`.
+
+### Changed
+
+- `FeedSpec.price_col` now drives the broker reference price instead of being collapsed back to `close`.
+- Market execution can use side-aware quotes: buys at ask, sells at bid.
+- `QUOTE_SIDE` marking prices long inventory on the bid and short inventory on the ask.
+- Result persistence now includes fills alongside trades, equity, daily P&L, metrics, and config.
+- Result persistence now includes portfolio state alongside trades, fills, equity,
+  daily P&L, metrics, and config.
+- User guides and README now document quote-aware feeds, mark pricing, and fill export.
+- User guides and README now document portfolio-state reporting and quote-aware audit fields.
+
+### Performance
+
+- Legacy OHLCV hot path remains faster than the pre-optimization baseline.
+- Quote-aware execution adds moderate overhead relative to the optimized OHLCV path, while staying ahead of the legacy baseline in local benchmarks.
+
+### Validation
+
+- `uv run ty check`
+- `pre-commit run --all-files`
+- `uv run pytest tests/ -q`

README.md

@@ -21,9 +21,11 @@ Backtesting requires accurate simulation of order execution, position tracking,
 - Event-driven architecture with point-in-time correctness (no look-ahead bias)
 - Exit-first order processing matching real broker behavior
 - Configurable execution modes (same-bar or next-bar fills)
+- Quote-aware execution and marking with `price`, bid, ask, midpoint, and side-aware sources
 - Position-level risk rules (stop-loss, take-profit, trailing stops)
 - Portfolio-level constraints (max positions, drawdown limits)
 - Cash, margin, and crypto account policies
+- First-class trade, fill, and portfolio-state export for audit and downstream analysis
 - 40+ behavioral knobs for framework-specific parity
 
 The same Strategy class used in backtesting works unchanged in ml4t-live for production deployment.
@@ -46,7 +48,7 @@ class SignalStrategy(Strategy):
     def on_data(self, timestamp, data, context, broker):
         for asset, bar in data.items():
             signal = bar.get("signals", {}).get("prediction", 0)
-            price = bar.get("close", 0)
+            price = bar.get("price", bar.get("close", 0))
             position = broker.get_position(asset)
 
             if position is None and signal > 0.5:
@@ -68,8 +70,11 @@ result = engine.run()
 
 print(f"Total Return: {result.metrics['total_return_pct']:.2f}%")
 print(f"Sharpe Ratio: {result.metrics['sharpe']:.2f}")
+print(result.to_fills_dataframe().head())
 ```
 
+`bar["price"]` follows `FeedSpec.price_col` when you provide one, so the same strategy works for close-based bars and quote-aware feeds.
+
 ## Risk Management
 
 Position-level exit rules:
@@ -135,6 +140,59 @@ config = BacktestConfig(
 )
 ```
 
+## Quote-Aware Execution
+
+```python
+from ml4t.backtest import BacktestConfig, DataFeed
+from ml4t.backtest.config import ExecutionPrice
+
+feed = DataFeed(
+    prices_df=quotes,
+    price_col="mid_price",
+    bid_col="bid",
+    ask_col="ask",
+    bid_size_col="bid_size",
+    ask_size_col="ask_size",
+)
+
+config = BacktestConfig(
+    execution_price=ExecutionPrice.QUOTE_SIDE,
+    mark_price=ExecutionPrice.QUOTE_SIDE,
+)
+```
+
+With `QUOTE_SIDE`, buys fill at the ask and sells fill at the bid when quotes are present. `mark_price` is configured separately, so you can trade on one source and mark the book on another.
+
+Quote-aware runs also preserve the microstructure context in the result surface:
+
+- `result.to_fills_dataframe()` includes bid/ask/midpoint/spread/size context
+- `result.to_trades_dataframe()` includes nullable entry/exit quote summaries
+- `result.to_portfolio_state_dataframe()` reflects the configured mark source over time
+- `result.to_predictions_dataframe()` preserves the raw model/input surface for downstream
+  diagnostics
+
+## Reproducible Config Snapshots
+
+`BacktestConfig` is also the serializable backtest preset surface. You can keep
+input configs sparse, then persist the fully resolved config that actually ran.
+
+```python
+config = BacktestConfig.from_yaml("config/my_backtest.yaml")
+result = Engine(feed, strategy, config).run()
+
+resolved_config = result.config.to_dict()
+runtime_spec = result.to_spec_dict()
+written = result.to_parquet("results/run_001")
+```
+
+The exported result directory includes:
+
+- `config.yaml` for the replayable resolved config payload
+- `spec.yaml` for the richer runtime snapshot with library version and realized run window
+
+Use top-level `feed` in `BacktestConfig` for generic feed semantics and top-level
+`metadata` for user-defined provenance like input paths or strategy ids.
+
 ## Commission and Slippage
 
 ```python
@@ -234,12 +292,14 @@ Benchmark on 250 assets x 20 years daily data (1.26M bars):
 ## Documentation
 
 - [Getting Started](docs/getting-started/quickstart.md) — your first backtest
+- [Data Feed](docs/user-guide/data-feed.md) — `price_col`, quote columns, and feed wiring
 - [Strategies](docs/user-guide/strategies.md) — strategy interface and templates
 - [Stateful Strategies](docs/user-guide/stateful-strategies.md) — advanced event-driven patterns (Kelly sizing, pairs trading, circuit breakers)
 - [Execution Semantics](docs/user-guide/execution-semantics.md) — fill timing, ordering, stops
 - [Configuration](docs/user-guide/configuration.md) — 40+ behavioral knobs
 - [Risk Management](docs/user-guide/risk-management.md) — stops, trails, portfolio limits
 - [Rebalancing](docs/user-guide/rebalancing.md) — weight-based portfolio management
+- [Results & Analysis](docs/user-guide/results.md) — trades, fills, equity, and Parquet export
 - [Market Impact](docs/user-guide/market-impact.md) — commission, slippage, and impact models
 - [Profiles](docs/user-guide/profiles.md) — framework parity presets
 
@@ -248,7 +308,8 @@ Benchmark on 250 assets x 20 years daily data (1.26M bars):
 - **Event-driven**: Each bar processes sequentially with exit-first logic
 - **Point-in-time**: No access to future data within strategy callbacks
 - **Configurable fills**: Match behavior of different backtesting frameworks
-- **Parquet export**: Results serializable for analysis with ml4t-diagnostic
+- **Quote-aware**: Optional bid/ask/mid/size caches with side-aware market fills
+- **Parquet export**: Trades, fills, equity, daily P&L, and config are serializable
 - **Type-safe**: 0 type diagnostics (ty/Astral), full type annotations
 
 ## Related Libraries

docs/api/index.md

@@ -117,6 +117,7 @@ Auto-generated from source docstrings.
     options:
       show_root_heading: true
       members:
+        - to_predictions_dataframe
         - to_trades_dataframe
         - to_equity_dataframe
         - to_dict

docs/getting-started/quickstart.md

@@ -42,7 +42,7 @@ print(f"Trades:       {result.metrics['num_trades']}")
 
 ## Data Format
 
-DataFeed expects a Polars DataFrame with these columns:
+DataFeed expects a Polars DataFrame keyed by `timestamp` and `asset` plus at least one price column. Standard OHLCV is the default:
 
 | Column | Type | Required |
 |--------|------|----------|
@@ -57,6 +57,8 @@ DataFeed expects a Polars DataFrame with these columns:
 For multi-asset backtests, stack all assets in a single DataFrame -- the engine
 handles partitioning by timestamp automatically.
 
+`bar["price"]` is always populated. By default it follows `close`, but it switches to `FeedSpec.price_col` or the `price_col=` override when you provide one.
+
 ## Strategy Callbacks
 
 Every strategy subclasses `Strategy` and implements `on_data`:
@@ -72,7 +74,7 @@ class MyStrategy(Strategy):
 
         Args:
             timestamp: Current bar's datetime
-            data: Dict of {asset: {open, high, low, close, volume, signals}}
+            data: Dict of {asset: {price, open, high, low, close, volume, signals, ...}}
             context: Dict of context data (if provided)
             broker: Broker for submitting orders and querying positions
         """
@@ -99,7 +101,7 @@ class SignalStrategy(Strategy):
             if signal > 0.7 and position is None:
                 # Buy 10% of portfolio value
                 equity = broker.get_account_value()
-                shares = int(equity * 0.10 / bar["close"])
+                shares = int(equity * 0.10 / bar["price"])
                 if shares > 0:
                     broker.submit_order(asset, shares)
 
@@ -110,6 +112,31 @@ class SignalStrategy(Strategy):
 result = run_backtest(prices, SignalStrategy(), signals=signals_df)
 ```
 
+## Quote-Aware Feeds
+
+If you have quotes, add them without changing your strategy interface:
+
+```python
+from ml4t.backtest import BacktestConfig, DataFeed
+from ml4t.backtest.config import ExecutionPrice
+
+feed = DataFeed(
+    prices_df=quotes_df,
+    price_col="mid_price",
+    bid_col="bid",
+    ask_col="ask",
+    bid_size_col="bid_size",
+    ask_size_col="ask_size",
+)
+
+config = BacktestConfig(
+    execution_price=ExecutionPrice.QUOTE_SIDE,
+    mark_price=ExecutionPrice.QUOTE_SIDE,
+)
+```
+
+Buys then fill from the ask, sells fill from the bid, and `bar["price"]` still gives your configured reference price.
+
 ## Adding Transaction Costs
 
 ```python
@@ -167,7 +194,7 @@ class ProtectedStrategy(Strategy):
         for asset, bar in data.items():
             if broker.get_position(asset) is None:
                 equity = broker.get_account_value()
-                shares = int(equity * 0.10 / bar["close"])
+                shares = int(equity * 0.10 / bar["price"])
                 if shares > 0:
                     broker.submit_order(asset, shares)
 ```
@@ -193,10 +220,22 @@ print(trades_df.head())
 equity_df = result.to_equity_dataframe()
 print(equity_df.head())
 
+# Fills as Polars DataFrame
+fills_df = result.to_fills_dataframe()
+print(fills_df.head())
+
+# Portfolio state snapshots
+portfolio_df = result.to_portfolio_state_dataframe()
+print(portfolio_df.head())
+
 # Export to Parquet for analysis with ml4t-diagnostic
 result.to_parquet("./results/my_backtest")
 ```
 
+For quote-aware backtests, `fills_df` and `trades_df` preserve the quote context
+used for execution, while `portfolio_df` shows the effect of the configured
+marking source over time.
+
 ## Convenience Function
 
 For quick experiments, `run_backtest` combines DataFeed + Engine in one call:

docs/index.md

@@ -34,6 +34,8 @@ result = run_backtest(prices, BuyAndHold(), config="backtrader")
 
 **Configurable execution semantics.** Every behavioral difference between backtesting frameworks (fill ordering, stop modes, cash policies, settlement) is a named config parameter. Switch profiles to replicate any framework exactly.
 
+**Quote-aware when you need it.** The feed can cache bid, ask, midpoint, and quote sizes additively. Market execution and position marking can use `price`, `bid`, `ask`, `quote_mid`, or `quote_side`.
+
 **Validated at scale.** 225,000+ trades verified trade-by-trade against VectorBT Pro, Backtrader, Zipline, and LEAN on real market data (250 assets x 20 years).
 
 **Fast.** 19x faster than Backtrader, 8x faster than Zipline, 5x faster than LEAN on identical workloads. Processes 40,000+ bars/second across 250 assets.
@@ -42,9 +44,11 @@ result = run_backtest(prices, BuyAndHold(), config="backtrader")
 |---------|-------------|
 | Event-driven | Point-in-time correctness, no look-ahead bias |
 | 40+ behavioral knobs | Every execution detail is configurable |
+| Quote-aware execution | Side-aware fills and separate mark pricing |
 | 10 framework profiles | Match VectorBT, Backtrader, Zipline, LEAN exactly |
 | Risk management | Stop-loss, take-profit, trailing stops, portfolio limits |
 | Multi-asset | Rebalancing, weight targets, exit-first ordering |
+| Rich persistence | Export trades, fills, equity, portfolio state, and daily P&L to Parquet |
 
 ## Parity Validation