fix: harden storage writes and update workflows

Author: Stefan Jansen

README.md

@@ -106,7 +106,7 @@ fred = FREDProvider().fetch_series("GDP", "2020-01-01", "2024-12-31")
 | EODHD | 60+ global exchanges |
 | Tiingo | US equities with quality focus |
 | Twelve Data | Multi-asset coverage |
-| Databento | CME, CBOE, ICE futures/options |
+| Databento | CME/ICE futures; OPRA available via native SDK |
 | Massive | US equities, options, futures, forex, crypto |
 | Finnhub | 70+ global exchanges |
 | Binance | Crypto exchange data |

docs/README.md

@@ -22,7 +22,7 @@ Complete documentation for the ML4T Data market data library.
 - **[Cryptocurrency Guide](asset-classes/crypto.md)** - CoinGecko, CryptoCompare (free tier available)
 - **[Equities Guide](asset-classes/equities.md)** - US & Global stocks, ETFs (7 providers)
 - **[Forex Guide](asset-classes/forex.md)** - OANDA, currency pairs, forex trading
-- **[Futures Guide](asset-classes/futures.md)** - Databento, derivatives, institutional data
+- **[Futures Guide](asset-classes/futures.md)** - Databento futures and institutional data
 
 ---
 
@@ -61,7 +61,7 @@ Complete documentation for the ML4T Data market data library.
 
 ### Provider-Specific
 
-- **[Databento Reference](providers/databento_reference.md)** - Futures and derivatives
+- **[Databento Reference](providers/databento_reference.md)** - Futures and native Databento datasets
 - **[Yahoo Finance](yahoo_provider.md)** - Free stock data (legacy)
 - **[Crypto Providers](crypto_providers.md)** - Cryptocurrency data sources
 
@@ -133,7 +133,7 @@ Quick comparison of all 14 providers:
 | **OANDA** | ❌ | ❌ | ✅ | ❌ | Demo | Professional forex |
 | **Twelve Data** | ✅ | ✅ | ✅ | ❌ | 800/day | Multi-asset |
 | **Polygon** | ✅ | ✅ | ✅ | ✅ | 5/min | Multi-asset (paid) |
-| **Databento** | ❌ | ⚠️ | ❌ | ✅ | None | Institutional futures |
+| **Databento** | ❌ | ⚠️ | ❌ | ⚠️ native SDK | None | Institutional futures |
 | **Yahoo** | ❌ | ✅ | ❌ | ❌ | Unlimited | Free stocks (unreliable) |
 | **Binance** | ✅ | ❌ | ❌ | ✅ | 1200/min | Crypto exchange data |
 
@@ -184,9 +184,9 @@ eurusd = provider.fetch_ohlcv("EUR_USD", "2024-01-01", "2024-12-31", frequency="
 ### Get Started with Futures
 
 ```python
-from ml4t.data.providers import DatabentoProvider
+from ml4t.data.providers import DataBentoProvider
 
-provider = DatabentoProvider(api_key="your_key", dataset="GLBX.MDP3")
+provider = DataBentoProvider(api_key="your_key", dataset="GLBX.MDP3")
 es = provider.fetch_ohlcv("ES.v.0", "2024-01-01", "2024-12-31")  # S&P 500 futures
 ```
 

docs/asset-classes/README.md

@@ -104,10 +104,10 @@ df = provider.fetch_ohlcv("EUR_USD", start="2024-01-01", end="2024-12-31", frequ
 
 ---
 
-### [📉 Futures & Options Data Guide](./futures.md)
+### [Futures Data Guide](./futures.md)
 
-**Asset Classes**: Futures Contracts, Options on Futures
-**Providers**: Databento (primary), Polygon
+**Asset Classes**: Futures Contracts
+**Providers**: Databento (primary), Massive for listed options
 **Difficulty**: 🔴 Advanced
 **Free Tier**: ❌ None (Databento requires paid subscription ~$30-50/month)
 
@@ -119,9 +119,9 @@ df = provider.fetch_ohlcv("EUR_USD", start="2024-01-01", end="2024-12-31", frequ
 
 **Quick start**:
 ```python
-from ml4t.data.providers import DatabentoProvider
+from ml4t.data.providers import DataBentoProvider
 
-provider = DatabentoProvider(api_key="your_key", dataset="GLBX.MDP3")
+provider = DataBentoProvider(api_key="your_key", dataset="GLBX.MDP3")
 df = provider.fetch_ohlcv("ES.v.0", start="2024-01-01", end="2024-12-31")  # S&P 500 futures
 ```
 
@@ -191,7 +191,7 @@ df = provider.fetch_ohlcv("ES.v.0", start="2024-01-01", end="2024-12-31")  # S&P
 | **OANDA** | ❌ | ❌ | ✅ | ❌ | Demo account | Required | Professional forex |
 | **Twelve Data** | ✅ | ✅ | ✅ | ❌ | 800/day | Required | Multi-asset |
 | **Polygon** | ✅ | ✅ | ✅ | ✅ | 5/min | Required | Multi-asset (paid) |
-| **Databento** | ❌ | ⚠️ | ❌ | ✅ | None | Required | Institutional futures |
+| **Databento** | ❌ | ⚠️ | ❌ | ⚠️ native SDK | None | Required | Institutional futures |
 
 **Legend**:
 - ✅ Full support

docs/asset-classes/futures.md

@@ -1,18 +1,19 @@
-# Futures & Options Data Guide
+# Futures Data Guide
 
-**Asset Classes**: Futures contracts, Options on futures
-**Available Providers**: Databento (primary), Polygon
+**Asset Classes**: Futures contracts
+**Available Providers**: Databento (primary), Massive for listed options
 **Difficulty**: Advanced (requires understanding of derivatives mechanics)
 **Recommended For**: Institutional strategies, derivatives research, quantitative trading
 
 ---
 
 ## Overview
 
-ML4T Data provides access to **institutional-grade futures and options data** through Databento, the gold standard for derivatives market data. Futures markets have unique complexities:
+ML4T Data provides access to institutional-grade futures data through Databento.
+Futures markets have unique complexities:
 
 - **Expiration & Rolling**: Contracts expire monthly/quarterly, requiring continuous contract construction
-- **Multiple Exchanges**: CME, CBOE, ICE, Eurex - each with different conventions
+- **Multiple Exchanges**: CME, ICE, Eurex, and others each have different conventions
 - **Session Times**: Futures trade nearly 24 hours with complex session breaks
 - **Tick Data**: Ultra-high-frequency data (microsecond timestamps)
 - **Margin Requirements**: Leveraged instruments with daily mark-to-market
@@ -21,10 +22,12 @@ ML4T Data provides access to **institutional-grade futures and options data** th
 
 | Provider | Coverage | Free Tier | Data Quality | Best For |
 |----------|----------|-----------|--------------|----------|
-| **Databento** | CME, CBOE, ICE + more | ❌ Paid only | ⭐⭐⭐⭐⭐ Institutional | Professional futures trading |
+| **Databento** | CME/ICE/Eurex futures; OPRA via native SDK | Paid only | Institutional | Professional futures trading |
 | **Polygon** | Limited futures | 5/min | ⭐⭐⭐ Good | Multi-asset portfolios |
 
-**Important**: Futures data is **NOT free**. Databento requires paid subscription (~$30-50/month minimum for historical data).
+**Important**: Futures data is not free. Databento requires a paid subscription
+for most historical workflows. First-class Databento OPRA option-chain and quote
+helpers are planned but not yet implemented in ml4t-data.
 
 ---
 
@@ -36,19 +39,19 @@ ML4T Data provides access to **institutional-grade futures and options data** th
 - ✅ **Normalized data** - Consistent schemas across all exchanges
 - ✅ **Continuous contracts** - Automatic front-month rolling (.v.0 notation)
 - ✅ **Tick-level precision** - Microsecond timestamps
-- ✅ **CME, CBOE, ICE, Eurex** - Major derivatives exchanges worldwide
-- ✅ **Multiple schemas** - OHLCV, trades, quotes, market-by-order
+- ✅ **CME, ICE, Eurex** - Major futures exchanges worldwide
+- ✅ **Selected schemas** - OHLCV plus direct access to native Databento schemas
 
 **Best for**: Quantitative futures strategies, high-frequency trading, institutional research
 
 ### Quick Start
 
 ```python
-from ml4t.data.providers import DatabentoProvider
+from ml4t.data.providers import DataBentoProvider
 
 # Get API key from: https://databento.com/
 # NOTE: Requires paid subscription ($30-50/month minimum)
-provider = DatabentoProvider(
+provider = DataBentoProvider(
     api_key="your_databento_api_key",  # Or set DATABENTO_API_KEY env var
     dataset="GLBX.MDP3"  # CME Globex futures (default)
 )
@@ -191,31 +194,33 @@ Databento provides data from multiple exchanges:
 
 ```python
 # CME Globex (most futures)
-provider = DatabentoProvider(dataset="GLBX.MDP3")
+provider = DataBentoProvider(dataset="GLBX.MDP3")
 
 # CME (all CME markets)
-provider = DatabentoProvider(dataset="CME.MDP3")
+provider = DataBentoProvider(dataset="CME.MDP3")
 
-# CBOE Options (equity options)
-provider = DatabentoProvider(dataset="OPRA.PILLAR")
+# OPRA options through the native client for advanced workflows
+provider = DataBentoProvider(dataset="OPRA.PILLAR")
+client = provider.client
 
 # US Equities
-provider = DatabentoProvider(dataset="XNAS.ITCH")  # NASDAQ
-provider = DatabentoProvider(dataset="XNYS.TRADES")  # NYSE
+provider = DataBentoProvider(dataset="XNAS.ITCH")  # NASDAQ
+provider = DataBentoProvider(dataset="XNYS.TRADES")  # NYSE
 ```
 
 **Available datasets**:
 - `GLBX.MDP3` - CME Globex (futures, options on futures)
 - `CME.MDP3` - CME (all markets)
 - `XCME.MDP3` - CME Crypto
-- `OPRA.PILLAR` - CBOE Equity Options
+- `OPRA.PILLAR` - OPRA listed options (native client; first-class helpers planned)
 - `XNAS.ITCH` - NASDAQ Equities
 - `XNYS.TRADES` - NYSE Equities
 - `BATS.PITCH` - BATS Equities
 
 ### Schemas (Data Types)
 
-Databento supports multiple data schemas beyond OHLCV:
+Databento supports multiple data schemas beyond OHLCV. The wrapper exposes
+selected schema fetches and the native client for advanced workflows:
 
 ```python
 # OHLCV aggregations
@@ -234,14 +239,14 @@ Databento supports multiple data schemas beyond OHLCV:
 **Example: Fetching tick data**:
 
 ```python
-provider = DatabentoProvider(dataset="GLBX.MDP3", default_schema="trades")
+provider = DataBentoProvider(dataset="GLBX.MDP3")
 
 # Fetch trade-by-trade data for ES
 trades = provider.fetch_ohlcv(
     symbol="ES.v.0",
     start="2024-11-04",
     end="2024-11-04",
-    frequency="tick"  # Uses default_schema="trades"
+    frequency="tick"  # requests Databento trades schema
 )
 
 # Result: Every single trade with microsecond timestamps
@@ -288,7 +293,7 @@ Futures markets trade nearly 24 hours with session breaks:
 # But in UTC: Monday 00:00 - Tuesday 22:00
 
 # Databento handles this with adjust_session_dates
-provider = DatabentoProvider(
+provider = DataBentoProvider(
     dataset="GLBX.MDP3",
     adjust_session_dates=True,  # Align dates with session start
     session_start_hour_utc=22  # 22:00 UTC = 5:00 PM CT (CST)
@@ -310,16 +315,16 @@ for symbol in symbols:
     # 50 symbols = 50 requests = completes in 0.5 seconds
 ```
 
-**Live WebSocket**: Unlimited (within subscription limits)
+**Live WebSocket**: Use Databento's native SDK directly.
 
 ### Complete Example: Multi-Contract Strategy
 
 ```python
-from ml4t.data.providers import DatabentoProvider
+from ml4t.data.providers import DataBentoProvider
 import polars as pl
 
 # Initialize provider
-provider = DatabentoProvider(
+provider = DataBentoProvider(
     dataset="GLBX.MDP3",
     adjust_session_dates=True
 )
@@ -375,7 +380,7 @@ except DataNotAvailableError as e:
 
 try:
     # Using expired API key
-    provider = DatabentoProvider(api_key="expired_key")
+    provider = DataBentoProvider(api_key="expired_key")
 except AuthenticationError as e:
     print(f"Authentication failed: {e}")
     # Renew subscription or check API key
@@ -579,7 +584,7 @@ annual_roll_cost = roll_cost_dollars * annual_rolls * num_contracts
 # Daily break: 4:00 PM - 5:00 PM CT
 
 # Use adjust_session_dates for proper date alignment
-provider = DatabentoProvider(
+provider = DataBentoProvider(
     adjust_session_dates=True,
     session_start_hour_utc=22  # 5:00 PM CT = 22:00 UTC (CST)
 )

docs/getting-started/installation.md

@@ -30,7 +30,7 @@ Some providers require additional dependencies:
 # Yahoo Finance (included by default)
 pip install "ml4t-data[yahoo]"
 
-# DataBento (institutional futures/options)
+# DataBento (institutional futures)
 pip install "ml4t-data[databento]"
 
 # OANDA (forex)

docs/getting-started/provider-selection.md

@@ -73,7 +73,7 @@ flowchart TD
 | **Finnhub** | ✅ | ✅ | ✅ | ✅ | ❌ | Yes | Paid only | Professional grade |
 | **Twelve Data** | ✅ | ✅ | ❌ | ✅ | ❌ | Yes | 800/day | Multi-asset + indicators |
 | **OANDA** | ❌ | ❌ | ❌ | ✅ | ❌ | Yes | No | Professional forex |
-| **Databento** | ❌ | ✅ | ❌ | ❌ | ✅ | Yes | Paid only | Institutional derivatives |
+| **Databento** | ❌ | ✅ | ❌ | ❌ | ✅ | Yes | Paid only | Institutional futures |
 | **Polygon** | ✅ | ✅ | ❌ | ✅ | ❌ | Yes | Paid only | Professional multi-asset |
 
 ### By Pricing
@@ -105,7 +105,7 @@ flowchart TD
 |----------|----------------|--------------|
 | **Finnhub** | $59.99/mo | Historical OHLCV, 70+ exchanges |
 | **Polygon** | $99/mo | Multi-asset, professional-grade |
-| **Databento** | Custom | Institutional derivatives data |
+| **Databento** | Custom | Institutional futures data |
 
 ## Decision Guidelines
 
@@ -125,7 +125,7 @@ flowchart TD
 3. **Finnhub** - Real-time + historical, professional grade
 
 ### For Institutions
-1. **Databento** - Tick-level derivatives data
+1. **Databento** - Tick-level futures data
 2. **Polygon** - Multi-asset professional data
 3. **Finnhub** - Global exchange coverage
 

docs/provider-selection-guide.md

@@ -73,7 +73,7 @@ flowchart TD
 | **Finnhub** | ✅ | ✅ | ✅ | ✅ | ❌ | Yes | Paid only | Professional grade |
 | **Twelve Data** | ✅ | ✅ | ❌ | ✅ | ❌ | Yes | 800/day | Multi-asset + indicators |
 | **OANDA** | ❌ | ❌ | ❌ | ✅ | ❌ | Yes | No | Professional forex |
-| **Databento** | ❌ | ✅ | ❌ | ❌ | ✅ | Yes | Paid only | Institutional derivatives |
+| **Databento** | ❌ | ✅ | ❌ | ❌ | ✅ | Yes | Paid only | Institutional futures |
 | **Polygon** | ✅ | ✅ | ❌ | ✅ | ❌ | Yes | Paid only | Professional multi-asset |
 
 ### By Pricing
@@ -105,7 +105,7 @@ flowchart TD
 |----------|----------------|--------------|
 | **Finnhub** | $59.99/mo | Historical OHLCV, 70+ exchanges |
 | **Polygon** | $99/mo | Multi-asset, professional-grade |
-| **Databento** | Custom | Institutional derivatives data |
+| **Databento** | Custom | Institutional futures data |
 
 ## Decision Guidelines
 
@@ -125,7 +125,7 @@ flowchart TD
 3. **Finnhub** - Real-time + historical, professional grade
 
 ### For Institutions
-1. **Databento** - Tick-level derivatives data
+1. **Databento** - Tick-level futures data
 2. **Polygon** - Multi-asset professional data
 3. **Finnhub** - Global exchange coverage
 

docs/providers/PROVIDER_AUDIT.md

@@ -107,7 +107,7 @@ This document provides a comprehensive audit of all 21 data providers in ml4t-da
 | TBBO (Top of book) | Yes | MEDIUM |
 | Trades (tick) | Yes | MEDIUM |
 | WebSocket streaming | Yes | NOT SUPPORTING |
-| OPRA options | Yes | HIGH |
+| OPRA options | Yes | HIGH - planned first-class support |
 | Symbology API | Yes | LOW |
 
 #### Coverage
@@ -391,7 +391,8 @@ This document provides a comprehensive audit of all 21 data providers in ml4t-da
 | Quick start (free) | YahooFinance |
 | US equities (production) | EODHD or Massive |
 | Global equities | EODHD |
-| Futures/options | Databento |
+| Futures | Databento |
+| Listed options | Massive; Databento OPRA support planned |
 | Crypto | BinancePublic |
 | Forex | Oanda |
 | Academic factors | Fama-French + AQR |

docs/providers/README.md

@@ -75,7 +75,7 @@ For detailed pricing, terms, and gap analysis, see [PROVIDER_AUDIT.md](PROVIDER_
 |----------|--------|--------|-------|---------|--------------|
 | YahooFinance | ✅ (7d) | ✅ | ✅ | ❌* | ❌* |
 | Alpaca | ✅ | ✅ | ✅ | ❌ | ❌ |
-| Databento | ✅ | ✅ | ✅ | ✅ (OPRA) | ❌ |
+| Databento | ✅ | ✅ | ✅ | Planned (OPRA) | ❌ |
 | Massive | ✅ | ✅ | ✅ | ✅ | ✅ |
 | EODHD | ❌ | ❌ | ✅ | ✅ ($29.99) | ✅ ($59.99) |
 | Tiingo | ✅ | ✅ | ✅ | ❌ | ❌ |
@@ -177,7 +177,8 @@ print(f"Total history: {len(combined)} trading days")
 | Quick start (free) | YahooFinance |
 | US equities (production) | EODHD or Massive |
 | Global equities | EODHD |
-| Futures/options | Databento |
+| Futures | Databento |
+| Listed options | Massive; Databento OPRA support planned |
 | Crypto | BinancePublic |
 | Forex | Oanda |
 | Academic factors | Fama-French + AQR |