Skip to content

Commit a9beec1

Browse files
stefan-jansenclaude
andcommitted
refactor: Complete library renaming from qdata to ml4t-data
Major refactoring to align library naming with sibling libraries (ml4t-features, ml4t-eval, ml4t-backtest) and add corporate actions educational materials. **Library Renaming (qdata → ml4t-data)**: - Package name: `ml4t-data` (for pip install) - Python imports: `ml4t.data` (for code) - Class names: `QDataConfig` → `DataConfig` - Updated 1,100+ references across all files - Deleted: `docs/extending_qdata.md` - Added: `docs/extending_ml4t_data.md` **Corporate Actions Module**: - Location: `src/ml4t/data/adjustments/core.py` - Industry-standard backward adjustment formula - Adaptive reverse split detection algorithm - 70% validation success rate on 100 random stocks - Comprehensive investigation documented in: - `CORPORATE_ACTIONS_FINAL_REPORT.md` - `YAHOO_QUANDL_FEASIBILITY.md` - `CORPORATE_ACTIONS_FINDINGS.md` **Educational Demo Project**: - Location: `~/ml3t/data/equities/corporate_actions_demo/` - Purpose: ML4T book chapter on data quality challenges - Demonstrates real-world issues with free data sources - Includes Python script and Jupytext notebook format - Documents Yahoo Finance vs Quandl incompatibilities **Code Quality Improvements**: - Fixed all pre-commit hook violations (27 errors) - F821: Added TYPE_CHECKING import for DataObject in hive.py - E712: Replaced boolean == True/False with idiomatic checks - F841: Removed unused variables in example scripts - F401: Fixed aiofiles import using importlib.util.find_spec - B007: Renamed unused loop variables with _ prefix - SIM108: Simplified if-else to ternary operators - Auto-formatted 28 files with ruff **Test Results**: - ✅ 849 tests passed - ⏭️ 23 skipped (flaky Yahoo tests in parallel execution) - ⏱️ Total time: 29 minutes 12 seconds - 🎯 Exit code: 0 **Documentation**: - Added comprehensive validation reports - Educational README for corporate actions demo - Extensive feasibility analysis of Yahoo/Quandl integration - Project summary and completion status 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 8e0fbe1 commit a9beec1

198 files changed

Lines changed: 31027 additions & 957 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/analysis/CRITICAL_TEST_FIXES.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
Method doesn't handle lowercase or mixed-case symbols correctly. Causes ~20-30 test failures.
1414

1515
### File
16-
`/home/stefan/ml4t/software/data/src/qdata/providers/coingecko.py`
16+
`/home/stefan/ml4t/software/data/src/ml4t.data/providers/coingecko.py`
1717

1818
### Current Code (Lines 376-394)
1919
```python
@@ -118,7 +118,7 @@ oanda = [
118118
"oandapyV20>=0.7.0",
119119
]
120120
all-providers = [
121-
"qdata[yahoo,databento,oanda]",
121+
"ml4t.data[yahoo,databento,oanda]",
122122
]
123123

124124
# Development dependencies
@@ -148,7 +148,7 @@ api = [
148148
]
149149

150150
all-providers = [
151-
"qdata[yahoo,databento,oanda]",
151+
"ml4t.data[yahoo,databento,oanda]",
152152
]
153153

154154
# Development dependencies
@@ -161,7 +161,7 @@ dev = [
161161

162162
# All optional dependencies
163163
all = [
164-
"qdata[all-providers,dev,docs,api]", # ← Update to include 'api'
164+
"ml4t.data[all-providers,dev,docs,api]", # ← Update to include 'api'
165165
]
166166
```
167167

@@ -197,7 +197,7 @@ python_functions = ["test_*"]
197197
addopts = [
198198
"-ra",
199199
"--strict-markers",
200-
"--cov=qdata",
200+
"--cov=ml4t.data",
201201
"--cov-report=term-missing",
202202
"--cov-report=html",
203203
]
@@ -258,7 +258,7 @@ pytest -m ""
258258
pytest -m "integration"
259259

260260
# Run with coverage (unit tests only)
261-
pytest --cov=qdata --cov-report=html
261+
pytest --cov=ml4t.data --cov-report=html
262262
```
263263

264264
---
@@ -408,7 +408,7 @@ pytest -x
408408
pytest -s
409409

410410
# Generate coverage report
411-
pytest --cov=qdata --cov-report=html
411+
pytest --cov=ml4t.data --cov-report=html
412412
open htmlcov/index.html
413413
```
414414

@@ -489,7 +489,7 @@ pytest tests/test_api/test_main.py
489489

490490
| File | Change | Impact |
491491
|------|--------|--------|
492-
| `src/qdata/providers/coingecko.py` | Fix symbol_to_id method | 20-30 tests fixed |
492+
| `src/ml4t.data/providers/coingecko.py` | Fix symbol_to_id method | 20-30 tests fixed |
493493
| `pyproject.toml` | Add `api` optional deps | API tests can run |
494494
| `pyproject.toml` | Update pytest config | Default runs fast |
495495
| `tests/test_yahoo_provider.py` | Add integration marker | Skipped by default |

.claude/analysis/TEST_AUDIT_EXECUTIVE_SUMMARY.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# QData Test Suite - Executive Summary
1+
# ML4T Data Test Suite - Executive Summary
22

33
**Date**: 2025-11-14
44
**Current Status**: 149 failed, 662 passed, 190 skipped, 6 errors
@@ -95,7 +95,7 @@ api = ["slowapi>=0.1.8"]
9595
## The Fix (30 minutes)
9696

9797
### Step 1: Fix CoinGecko Bug (5 minutes) 🔴
98-
**File**: `src/qdata/providers/coingecko.py:376`
98+
**File**: `src/ml4t.data/providers/coingecko.py:376`
9999
**Action**: Replace `symbol_to_id()` method with case-insensitive version
100100
**Result**: 20-30 tests fixed
101101

@@ -167,7 +167,7 @@ pytest -m "" # Run everything
167167

168168
| File | Lines Changed | Type |
169169
|------|---------------|------|
170-
| `src/qdata/providers/coingecko.py` | ~20 | Bug fix |
170+
| `src/ml4t.data/providers/coingecko.py` | ~20 | Bug fix |
171171
| `pyproject.toml` | ~10 | Config |
172172
| `tests/test_yahoo_provider.py` | 2 | Marker |
173173
| `tests/test_binance.py` | 2 | Marker |

.claude/analysis/TEST_FIX_CHECKLIST.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
## ✅ Critical Fixes (Do First - 15 minutes)
99

1010
### [ ] Fix 1: CoinGecko symbol_to_id Bug (5 min)
11-
**File**: `src/qdata/providers/coingecko.py`
11+
**File**: `src/ml4t.data/providers/coingecko.py`
1212
**Line**: 376
1313
**Action**: Replace entire `symbol_to_id` method
1414

@@ -55,7 +55,7 @@ api = [
5555
**And update** `all` section:
5656
```toml
5757
all = [
58-
"qdata[all-providers,dev,docs,api]", # ← Add 'api'
58+
"ml4t.data[all-providers,dev,docs,api]", # ← Add 'api'
5959
]
6060
```
6161

@@ -176,14 +176,14 @@ pytest -m integration
176176
pytest tests/integration/test_coingecko.py
177177

178178
# With coverage
179-
pytest --cov=qdata --cov-report=html
179+
pytest --cov=ml4t.data --cov-report=html
180180
```
181181

182182
---
183183

184184
## 📁 Files to Change
185185

186-
- [ ] `src/qdata/providers/coingecko.py` (bug fix)
186+
- [ ] `src/ml4t.data/providers/coingecko.py` (bug fix)
187187
- [ ] `pyproject.toml` (config + dependencies)
188188
- [ ] `tests/test_yahoo_provider.py` (marker)
189189
- [ ] `tests/test_binance.py` (marker)
@@ -220,7 +220,7 @@ pytest --help | grep "\-m"
220220
### "CoinGecko tests still failing"
221221
Verify method was replaced:
222222
```bash
223-
grep -A 10 "def symbol_to_id" src/qdata/providers/coingecko.py
223+
grep -A 10 "def symbol_to_id" src/ml4t.data/providers/coingecko.py
224224
# Should see: symbol_upper = symbol.upper()
225225
```
226226

.claude/analysis/TEST_SUITE_COMPREHENSIVE_AUDIT.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# QData Test Suite - Comprehensive Audit & Fix Plan
1+
# ML4T Data Test Suite - Comprehensive Audit & Fix Plan
22

33
**Date**: 2025-11-14
44
**Auditor**: Test Engineer Agent
@@ -9,7 +9,7 @@
99

1010
## Executive Summary
1111

12-
The qdata test suite is in **poor condition** requiring immediate systematic cleanup:
12+
The ml4t.data test suite is in **poor condition** requiring immediate systematic cleanup:
1313

1414
- **Primary Issues**: Missing dependencies (slowapi, optional providers), excessive integration test runtime, known bugs in provider code
1515
- **Collection Errors**: 6 errors blocking test discovery (already partially fixed via skip markers)
@@ -104,7 +104,7 @@ Based on code analysis, failures fall into these categories:
104104

105105
#### A1. CoinGecko `symbol_to_id()` Bug (HIGH PRIORITY)
106106

107-
**File**: `src/qdata/providers/coingecko.py:376`
107+
**File**: `src/ml4t.data/providers/coingecko.py:376`
108108
**Bug**: Method doesn't handle lowercase/mixed-case symbols correctly
109109

110110
**Current Implementation**:
@@ -243,7 +243,7 @@ api = [
243243
"slowapi>=0.1.8", # ❌ MISSING - ADD THIS
244244
]
245245
all-providers = [
246-
"qdata[yahoo,databento,oanda,api]",
246+
"ml4t.data[yahoo,databento,oanda,api]",
247247
]
248248
```
249249

@@ -269,7 +269,7 @@ all-providers = [
269269

270270
### Fix #1: CoinGecko symbol_to_id Bug (CRITICAL)
271271

272-
**File**: `/home/stefan/ml4t/software/data/src/qdata/providers/coingecko.py`
272+
**File**: `/home/stefan/ml4t/software/data/src/ml4t.data/providers/coingecko.py`
273273
**Line**: 376-394
274274

275275
**Current Code**:
@@ -343,7 +343,7 @@ api = [
343343

344344
# Update all-providers
345345
all = [
346-
"qdata[all-providers,dev,docs,api]",
346+
"ml4t.data[all-providers,dev,docs,api]",
347347
]
348348
```
349349

@@ -537,7 +537,7 @@ pip install -e ".[all]"
537537
pytest --maxfail=0 --tb=short
538538

539539
# Generate coverage report
540-
pytest --cov=qdata --cov-report=html --cov-report=term
540+
pytest --cov=ml4t.data --cov-report=html --cov-report=term
541541
```
542542

543543
---
@@ -547,7 +547,7 @@ pytest --cov=qdata --cov-report=html --cov-report=term
547547
### 🔴 CRITICAL (Do First - 30 minutes)
548548

549549
1. **Fix CoinGecko symbol_to_id bug**
550-
- File: `src/qdata/providers/coingecko.py:376`
550+
- File: `src/ml4t.data/providers/coingecko.py:376`
551551
- Replace method with fixed version (see Fix #1)
552552
- Run: `pytest tests/integration/test_coingecko.py::TestCoinGeckoProvider::test_symbol_to_id_mapping -v`
553553
- Expected: 3-4 tests now passing
@@ -605,7 +605,7 @@ pytest --cov=qdata --cov-report=html --cov-report=term
605605
- Update: Import paths if needed
606606

607607
11. **Add test coverage reporting**
608-
- Run: `pytest --cov=qdata --cov-report=html`
608+
- Run: `pytest --cov=ml4t.data --cov-report=html`
609609
- Identify: Untested code paths
610610
- Add: Unit tests for uncovered critical code
611611

@@ -688,7 +688,7 @@ pytest --cov=qdata --cov-report=html --cov-report=term
688688
## Part 12: Files Requiring Changes
689689

690690
### Code Fixes
691-
-`src/qdata/providers/coingecko.py` - Fix symbol_to_id method
691+
-`src/ml4t.data/providers/coingecko.py` - Fix symbol_to_id method
692692

693693
### Configuration Files
694694
-`pyproject.toml` - Add slowapi, update pytest config
@@ -706,7 +706,7 @@ pytest --cov=qdata --cov-report=html --cov-report=term
706706

707707
## Conclusion
708708

709-
The qdata test suite requires **systematic cleanup**, not ad-hoc fixes. The primary issues are:
709+
The ml4t.data test suite requires **systematic cleanup**, not ad-hoc fixes. The primary issues are:
710710

711711
1. **Critical Bug**: CoinGecko symbol_to_id doesn't handle case-insensitive symbols
712712
2. **Poor Organization**: Unit and integration tests mixed together

0 commit comments

Comments
 (0)