ml4t · stefan-jansen · Mar 3, 2026 · Mar 3, 2026 · Mar 3, 2026 · Mar 3, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -72,19 +72,23 @@ jobs:
       - name: Set up Python ${{ matrix.python-version }}
         run: uv python install ${{ matrix.python-version }}
 
+      - name: Install TA-Lib C library
+        run: |
+          wget -q http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
+          tar -xzf ta-lib-0.4.0-src.tar.gz
+          cd ta-lib/
+          ./configure --prefix=/usr/local
+          make
+          sudo make install
+          sudo ldconfig
+
       - name: Install dependencies
-        run: uv sync --dev
+        run: uv sync --dev --extra ta
 
       - name: Run tests
         run: |
           set +e
-          uv run pytest tests/ -v --tb=short -x --no-cov \
-            --deselect tests/core/test_calendars.py::TestNextOpenPreviousClose::test_next_open_from_market_hours \
-            --deselect tests/core/test_calendars.py::TestNextOpenPreviousClose::test_previous_close_from_market_hours \
-            --deselect tests/core/test_calendars.py::TestNextOpenPreviousClose::test_previous_close_from_before_open \
-            --deselect tests/core/test_calendars.py::TestNextOpenPreviousClose::test_previous_close_skips_weekend \
-            --deselect tests/core/test_calendars.py::TestSessionsBetween::test_sessions_between_same_week \
-            --deselect tests/core/test_calendars.py::TestCalendarIntegration::test_timezone_aware_workflow
+          uv run pytest tests/ -v --tb=short -x --no-cov
           PYTEST_EXIT=$?
           if [ $PYTEST_EXIT -eq 0 ] || [ $PYTEST_EXIT -eq 5 ]; then
             echo "Tests passed (exit code: $PYTEST_EXIT)"

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,130 @@
+# Changelog
+
+All notable changes to ml4t-engineer are documented in this file.
+
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Removed
+- Dead modules: `selection/`, `validation/`, `visualization/`, `pipeline/`
+- Diagnostic config classes (`feature_config.py`) — moved to ml4t-diagnostic
+- Deprecation machinery (`core/deprecation.py`, deprecated params in bar samplers and `mom()`)
+- Backward-compatibility shims in labeling module
+- `[tool.mypy]` config (migrated to ty)
+
+### Added
+- Comprehensive volatility tests (58 new tests covering all 11 non-TA-Lib estimators)
+- `perf` pytest marker — performance benchmarks excluded from default runs, available via `pytest -m perf`
+
+### Changed
+- TA-Lib moved from dev dependency group to `[ta]` optional extra (fixes CI for lint/typecheck jobs)
+- `mom()` parameter renamed: `timeperiod` → `period` (consistency with other indicators)
+- Bar sampler constructors: removed `initial_expectation` / `initial_run_expectation` params
+
+## [0.1.0a11] - 2026-03-03
+
+### Changed
+- API hardening and correctness fixes for beta preparation
+- Labeling leakage gap closed: data sorted chronologically before all label computations
+- Public API aligned with documentation
+
+## [0.1.0a10] - 2026-02-28
+
+### Fixed
+- Labeling leakage gap: ensured chronological sorting in all labeling functions
+- Public API documentation alignment
+
+## [0.1.0a9] - 2026-02-28
+
+### Fixed
+- `__version__` sourced from generated version metadata instead of hardcoded string
+
+## [0.1.0a8] - 2026-02-27
+
+### Added
+- GitHub Actions CI workflow (lint, typecheck, test matrix, build)
+- Release workflow with OIDC trusted publishing
+- Ecosystem diagrams in README
+
+### Changed
+- Removed outcome module (migrated to ml4t-diagnostic)
+- Feature count: 120 features across 10 categories
+- Standardized labeling API on `LabelingConfig`-first pattern
+
+### Fixed
+- Normalized metadata for 4 features (33 → 37 normalized)
+- ty type checking rules and CI configuration
+- Numba cleanup crash workaround for Python 3.13
+
+## [0.1.0a7] - 2026-01-20
+
+### Added
+- Time-based duration strings for labeling horizons (`"1h"`, `"4h"`, `"1d"`)
+- `fixed_time_horizon_labels()` accepts `horizon="1h"`
+- `triple_barrier_labels()` accepts `max_holding_period="1h"`
+- `rolling_percentile_binary_labels()` accepts time-based horizon/lookback
+- 51 new tests for time-based horizons
+
+### Fixed
+- Chronological sorting in `triple_barrier_labels`, `trend_scanning_labels`,
+  `fixed_time_horizon_labels`, and `rolling_percentile_binary_labels`
+- dtype-based timestamp detection (replaces name matching)
+
+## [0.1.0a6] - 2026-01-18
+
+### Added
+- Validation infrastructure with AFML and mlfinpy reference tests
+- 86 validation tests (AFML formulas + mlfinpy comparison)
+- Triple barrier, meta-labeling, sample weights validated at 1e-10 tolerance
+
+### Fixed
+- Triple barrier edge cases
+- Multiple drift detection bugs
+- Tuple syntax for isinstance type checks
+
+## [0.1.0a5] - 2026-01-14
+
+### Added
+- `get_agent_docs()` for AI agent discoverability
+- Hierarchical AGENT.md navigation files
+- AGENT.md files included in wheel builds
+
+### Fixed
+- `variance_ratio` Int64 bug
+
+## [0.1.0a4] - 2026-01-08
+
+### Fixed
+- Synced missing modules from development workspace
+
+## [0.1.0a3] - 2026-01-04
+
+Initial public alpha release.
+
+### Added
+- 120 feature functions across 10 categories (momentum, trend, volatility,
+  volume, microstructure, ML, risk, cycle, pattern, statistics)
+- 60 indicators validated against TA-Lib at 1e-6 tolerance
+- Triple-barrier labeling system (De Prado AFML)
+- ATR-adjusted barriers, fixed horizon, trend scanning, percentile labels
+- Meta-labeling and sample uniqueness (sequential bootstrap)
+- Alternative bar types: volume, dollar, tick, imbalance, run bars
+- Polars-native with Numba JIT compilation
+- `compute_features()` pipeline with dependency resolution
+- `FeatureCatalog` for feature discovery and metadata
+- `LabelingConfig` with Pydantic v2 serialization
+- `MLDatasetBuilder` for dataset construction
+- `PreprocessingPipeline` for feature transformation
+
+[Unreleased]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a11...HEAD
+[0.1.0a11]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a10...v0.1.0a11
+[0.1.0a10]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a9...v0.1.0a10
+[0.1.0a9]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a8...v0.1.0a9
+[0.1.0a8]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a7...v0.1.0a8
+[0.1.0a7]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a6...v0.1.0a7
+[0.1.0a6]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a5...v0.1.0a6
+[0.1.0a5]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a4...v0.1.0a5
+[0.1.0a4]: https://github.com/stefan-jansen/ml4t-engineer/compare/v0.1.0a3...v0.1.0a4
+[0.1.0a3]: https://github.com/stefan-jansen/ml4t-engineer/releases/tag/v0.1.0a3
diff --git a/README.md b/README.md
@@ -143,6 +143,17 @@ dbars = DollarBarSampler(dollar_threshold=1_000_000).sample(tick_data)
 ibars = TickImbalanceBarSampler(expected_imbalance=100).sample(tick_data)
 ```
 
+## Documentation
+
+- [Features](docs/user-guide/features.md) - 120 technical indicators across 11 categories
+- [Labeling](docs/user-guide/labeling.md) - 7 labeling methods for supervised learning
+- [Alternative Bars](docs/user-guide/bars.md) - Information-driven bar sampling
+- [Feature Discovery](docs/user-guide/discovery.md) - Registry, catalog, and search
+- [Fractional Differencing](docs/user-guide/fractional-differencing.md) - Memory-preserving stationarity
+- [ML-Readiness](docs/user-guide/ml-readiness.md) - Normalized features and preprocessing
+- [Preprocessing](docs/user-guide/preprocessing.md) - Scalers and leakage prevention
+- [Dataset Builder](docs/user-guide/dataset-builder.md) - Leakage-safe train/test preparation
+
 ## Technical Characteristics
 
 - **Polars-native**: All computations use Polars expressions

diff --git a/docs/audit/book-integration-audit.md b/docs/audit/book-integration-audit.md
@@ -0,0 +1,190 @@
+# Book Integration Audit: ml4t-engineer
+
+*Audit date: 2026-03-03 | Library version: v0.1.0a11 | Book: Machine Learning for Trading, 3rd Edition*
+
+## Executive Summary
+
+ml4t-engineer's core value proposition is validated by heavy book usage: 120 features, 7 labeling methods, and 11 bar samplers are all exercised in chapters 3, 7-9 and all 9 case studies. Several high-quality modules (MLDatasetBuilder, FeatureCatalog.search()) previously had zero book exposure but are now showcased in Ch7 NB10. The pipeline engine and DuckDB store are honestly low-value.
+
+---
+
+## Usage Matrix: Chapter x Module
+
+### Feature Computation (`compute_features`)
+
+| Chapter / Case Study | Notebook | Features Used | Notes |
+|---------------------|----------|---------------|-------|
+| Ch7 | `10_ml4t_library_ecosystem.py` | rsi, sma, ema, atr, macd, bollinger_bands | Registry tour, 3 input formats, MLDatasetBuilder |
+| Ch8 | `01_price_volume_features.py` | momentum (31), trend (10), volatility (15) | Core feature teaching |
+| Ch8 | `02_microstructure_features.py` | microstructure (12) | Kyle Lambda, VPIN, Amihud |
+| Ch8 | `03_structural_cross_instrument_features.py` | cross-asset (10) | beta_to_market, correlations |
+| Ch8 | `04_fundamentals_macro_calendar.py` | ML features, calendar | Lag, encodings, macro features |
+| Ch9 | `02_structural_breaks.py` | statistics | Structural break detection |
+| Ch9 | `03_fractional_differencing.py` | ffdiff (4) | Fractional differencing + ADF |
+| Ch9 | `05_spectral_features.py` | ML features | Spectral, FFT |
+| Ch9 | `08_garch_volatility.py` | volatility (15) | GARCH, EWMA, realized vol |
+| Ch9 | `09_har_rough_volatility.py` | volatility | HAR model features |
+| Ch9 | `11_hmm_regimes.py` | regime (6) | Hurst, HMM state probabilities |
+| Ch9 | `13_regime_as_feature.py` | regime (6) | Regime encoding as features |
+| Ch9 | `14_panel_features.py` | cross-asset (10) | Cross-sectional panel features |
+| ETFs | `03_features.py`, `04_temporal.py` | momentum, volatility, volume, ffdiff | Full pipeline |
+| US Equities Panel | `03_features.py`, `04_temporal.py` | momentum, volatility, ffdiff | Full pipeline |
+| CME Futures | `02_labels.py`, `03_features.py` | momentum, volatility, atr | Futures-specific |
+
+### Labeling Methods
+
+| Chapter / Case Study | Notebook | Method | Config Style |
+|---------------------|----------|--------|--------------|
+| Ch7 | `03_label_methods.py` | triple_barrier_labels | LabelingConfig.triple_barrier() |
+| Ch7 | `03_label_methods.py` | rolling_percentile_binary_labels | Direct call |
+| Ch7 | `03_label_methods.py` | trend_scanning_labels | Direct call |
+| Ch7 | `03_label_methods.py` | meta_labels + compute_bet_size | Meta-labeling workflow |
+| Ch7 | `03_label_methods.py` | sequential_bootstrap | Sample weighting |
+| CME Futures | `02_labels.py` | atr_triple_barrier_labels | LabelingConfig.atr_barrier() |
+| ETFs | `02_labels.py` | rolling_percentile_binary_labels | Direct call |
+| US Equities Panel | `02_labels.py` | triple_barrier_labels | LabelingConfig |
+| All case studies | `02_labels.py` | fixed_time_horizon_labels | Direct call |
+
+### Alternative Bar Sampling
+
+| Chapter | Notebook | Sampler | Notes |
+|---------|----------|---------|-------|
+| Ch3 | `08_itch_bar_sampling.py` | TickBarSampler, VolumeBarSampler, DollarBarSampler | ITCH tick data |
+| Ch3 | `10_itch_information_bars.py` | TickImbalanceBarSampler, FixedTickImbalanceBarSampler | Information-driven bars |
+| Ch3 | `13_databento_bar_sampling.py` | Bar sampling on Databento data | Alternative data source |
+
+### Feature Discovery & Registry
+
+| Chapter | Notebook | API Used |
+|---------|----------|----------|
+| Ch7 | `10_ml4t_library_ecosystem.py` | get_registry(), list_all(), get(), list_by_category() |
+| Ch7 | `10_ml4t_library_ecosystem.py` | feature_catalog.search(), feature_catalog.list(), describe() |
+| Ch7 | `10_ml4t_library_ecosystem.py` | compute_features (3 formats: list, dict, YAML) |
+
+### MLDatasetBuilder & Preprocessing
+
+| Chapter | Notebook | API Used |
+|---------|----------|----------|
+| Ch7 | `10_ml4t_library_ecosystem.py` | create_dataset_builder, train_test_split, scaler="robust" |
+| Ch7 | `10_ml4t_library_ecosystem.py` | LabelingConfig.to_yaml(), from_yaml() |
+| Ch7 | `02_preprocessing_pipeline.py` | StandardScaler, split-aware preprocessing |
+
+---
+
+## Book Chapter Structure (Actual)
+
+| Chapter | Directory | Notebooks | Primary ml4t-engineer Usage |
+|---------|-----------|-----------|----------------------------|
+| Ch3 | `03_market_microstructure/` | 17 | bars module |
+| Ch7 | `07_defining_learning_task/` | 10 | labeling, registry, dataset builder |
+| Ch8 | `08_feature_engineering/` | 8 + meta | features (all categories) |
+| Ch9 | `09_time_series_analysis/` | 14 + meta | volatility, regime, ffdiff, cross-asset |
+
+### Case Study Structure (Standard Pattern)
+
+All 9 case studies follow the same 18-file pattern:
+
+| Step | File | ml4t-engineer Usage |
+|------|------|---------------------|
+| Setup | `01_setup.py` | — |
+| Labels | `02_labels.py` | `atr_triple_barrier_labels`, `rolling_percentile_binary_labels`, `fixed_time_horizon_labels` |
+| Features | `03_features.py` | `compute_features`, individual feature functions |
+| Temporal | `04_temporal.py` | `ffdiff`, walk-forward CV |
+| Evaluation | `05_evaluation.py` | — (ml4t-diagnostic) |
+| Models | `06-13_*.py` | — |
+| Backtest | `14_backtest.py` | — (ml4t-backtest) |
+
+---
+
+## Feature Triage
+
+### Heavily Used (Core Value)
+
+| Module | Lines | Book Coverage | Confidence | Action |
+|--------|-------|---------------|------------|--------|
+| 120 features (10 categories) | ~8,000 | Ch8 (8 notebooks), Ch9 (14 notebooks), 9 case studies | 59 TA-Lib validated | Keep, document well |
+| 7 labeling methods | ~2,000 | Ch7 NB03, all 9 case study `02_labels.py` | AFML validated | Keep, document well |
+| 11 bar samplers | ~2,000 | Ch3 (3 notebooks) | Production-ready | Keep, document well |
+| ffdiff module | 383 | Ch9 NB03, ETFs/Equities `04_temporal.py` | Unique value | Keep, dedicated guide |
+| LabelingConfig | 467 | Ch7 NB03, all case studies | API surface | Keep, document well |
+| Registry/Catalog | ~650 | Ch7 NB10 | Discovery | Keep, dedicated guide |
+| MLDatasetBuilder | 638 | Ch7 NB10 (newly added) | Leakage-safe prep | Keep, dedicated guide |
+
+### Honestly Low-Value
+
+| Module | Lines | Assessment | Recommended Action |
+|--------|-------|------------|-------------------|
+| Pipeline engine | ~300 | `compute_features` already handles dependency ordering. Thin DAG wrapper adds little. | Label "Advanced" |
+| Store (DuckDB) | ~500 | No adoption path, no book usage, no clear user need. | Label "Experimental" |
+| FeatureSelector | stub | Correctly moved to ml4t-diagnostic. Stub remains as migration aid. | Keep stub, document redirect |
+
+---
+
+## Case Studies NOT Using ml4t-engineer
+
+These case studies implement features manually. This is **correct** in most cases:
+
+| Case Study | Reason for Manual Implementation | Library Overlap |
+|-----------|----------------------------------|-----------------|
+| Crypto Perps Funding | Domain-specific funding rate features | None — inline appropriate |
+| S&P 500 Options / Option Analytics | Greeks, IV surfaces — specialized derivatives analytics | None — out of scope |
+| US Firm Characteristics | Accounting ratios from financial statements | None — out of scope |
+| NASDAQ-100 Microstructure | Kyle's Lambda, Amihud, VPIN implemented manually for pedagogy | **High** — all in library (callout added) |
+| FX Pairs | Garman-Klass volatility, momentum features | **Partial** — some in library (callout added) |
+
+---
+
+## Cross-Reference: Book Notebooks Using ml4t.engineer
+
+### Direct imports (`from ml4t.engineer`)
+
+| File | Imports | Status |
+|------|---------|--------|
+| `07_defining_learning_task/code/10_ml4t_library_ecosystem.py` | compute_features, get_registry, feature_catalog, create_dataset_builder, LabelingConfig | Working |
+| `07_defining_learning_task/code/03_label_methods.py` | LabelingConfig, 7 labeling functions | Working (migrated from BarrierConfig, un-skipped) |
+| `07_defining_learning_task/code/04_minimum_favorable_adverse_excursion.py` | LabelingConfig | Working (migrated from BarrierConfig, un-skipped) |
+| `08_feature_engineering/code/01_price_volume_features.py` | ml4t.engineer.features.volatility, momentum, trend | Working |
+| `09_time_series_analysis/code/08_garch_volatility.py` | ml4t.engineer.features.volatility (6 functions) | Working |
+| All case study `02_labels.py` | ml4t.engineer.labeling (atr_triple_barrier_labels etc.) | Working |
+| All case study `03_features.py` | ml4t.engineer.features (momentum, volatility, regime, trend) | Working |
+
+### Indirect usage (via `utils/label_functions.py`)
+
+Some case study `02_labels.py` files use standalone label utility wrappers that mirror the ml4t.engineer API. These are isolated from API changes but are less idiomatic.
+
+---
+
+## Documentation Coverage
+
+| User Guide Page | Lines | Book Reference | Status |
+|----------------|-------|----------------|--------|
+| `labeling.md` | 522 | Ch7 `03_label_methods.py`, CME `02_labels.py`, ETFs `02_labels.py` | Complete |
+| `features.md` | 388 | Ch8 NB01-04, Ch9 NB08-14, ETFs/Equities/CME `03_features.py` | Complete |
+| `bars.md` | 405 | Ch3 `08_itch_bar_sampling.py`, `10_itch_information_bars.py`, `13_databento_bar_sampling.py` | Complete |
+| `ml-readiness.md` | 178 | Ch8 `01_price_volume_features.py` | Complete |
+| `discovery.md` | 162 | Ch7 `10_ml4t_library_ecosystem.py` | Complete |
+| `fractional-differencing.md` | 188 | Ch9 `03_fractional_differencing.py`, ETFs/Equities `04_temporal.py` | Complete |
+| `preprocessing.md` | 171 | Ch7 `02_preprocessing_pipeline.py` | Complete |
+| `dataset-builder.md` | 201 | Ch7 `10_ml4t_library_ecosystem.py` | Complete |
+
+---
+
+## Value Assessment
+
+### What ml4t-engineer does well
+
+1. **Feature computation is the clear winner**: 120 features, validated, fast, config-driven. Used in 30+ notebooks across 8 chapters and 9 case studies.
+2. **Labeling methods are comprehensive**: All 7 AFML methods implemented, validated, calendar-aware. Used in every case study.
+3. **Bar sampling is uniquely valuable**: No other Python library provides production-quality imbalance bars with threshold spiral warnings.
+4. **Registry/discovery is elegant**: Metadata-driven feature selection with TA-Lib compatibility flags and normalization status.
+5. **MLDatasetBuilder fills a real gap**: Leakage-safe dataset prep with CV integration — now demonstrated in Ch7 NB10.
+
+### What should be scoped honestly
+
+1. **Pipeline engine**: `compute_features` already does dependency ordering. The Pipeline class adds a thin DAG wrapper that few users need. Document as "Advanced".
+2. **DuckDB Store**: No user demand, no book usage. Keep but label experimental.
+3. **Cross-asset features (8 of 10 unused in book)**: Strong implementations but limited coverage. Only `beta_to_market` and `rolling_correlation` are commonly needed.
+
+---
+
+*This audit was used to drive the user guide expansion and book notebook updates for v0.1.0a11.*
diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md
@@ -97,6 +97,10 @@ print(info)
 
 ## Next Steps
 
-- [Features Guide](../user-guide/features.md) - Deep dive into all indicators
-- [Labeling Guide](../user-guide/labeling.md) - Triple-barrier and other labeling methods
+- [Features Guide](../user-guide/features.md) - 120 indicators across 11 categories
+- [Labeling Guide](../user-guide/labeling.md) - 7 labeling methods for supervised learning
+- [Alternative Bars](../user-guide/bars.md) - Information-driven bar sampling
+- [Feature Discovery](../user-guide/discovery.md) - Registry, catalog, and search API
+- [Fractional Differencing](../user-guide/fractional-differencing.md) - Memory-preserving stationarity
+- [Dataset Builder](../user-guide/dataset-builder.md) - Leakage-safe train/test preparation
 - [API Reference](../api/index.md) - Complete API documentation