feat: Make Delete Operator Strict, Improve Error Messages, Remove Color Formatting, and Make Local Raw References Lazy

.github/workflows/release.yml

@@ -0,0 +1,36 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - '*'
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    name: Create GitHub Release
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    if: startsWith(github.ref, 'refs/tags')
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Verify tag is on main branch
+        run: |
+          git fetch origin main
+          if ! git merge-base --is-ancestor ${{ github.sha }} origin/main; then
+            echo "Error: Tag is not on the main branch"
+            exit 1
+          fi
+
+      - name: Create Release
+        uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true
+          draft: false
+          prerelease: false

README.md

@@ -65,7 +65,9 @@ Sparkwheel builds on similar ideas but adds powerful features:
 |---------|-----------------|------------|
 | Config composition | Explicit (`+`, `++`) | **By default** (dicts merge, lists extend) |
 | Replace semantics | Default | Explicit with `=` operator |
-| Delete keys | Not idempotent | Idempotent `~` operator |
+| Delete keys | CLI-only `~` operator | `~` in **YAML and CLI** |
+| Delete list items | No ❌ | Yes ✅ (by index) |
+| Delete dict keys | CLI-only (`~foo.bar`) | Yes ✅ (YAML + CLI) |
 | References | OmegaConf interpolation | `@` (resolved) + `%` (raw YAML) |
 | Python expressions | Limited | Full Python with `$` |
 | Schema validation | Structured Configs | Python dataclasses |

docs/index.md

@@ -199,7 +199,7 @@ Sparkwheel has two types of references with distinct purposes:
     - **Composition-by-default** - Configs merge/extend naturally, no operators needed for common case
     - **List extension** - Lists extend by default (unique vs Hydra!)
     - **`=` replace operator** - Explicit control when you need replacement
-    - **`~` delete operator** - Remove inherited keys cleanly (idempotent!)
+    - **`~` delete operator** - Remove inherited keys explicitly
     - **Python expressions with `$`** - Compute values dynamically
     - **Dataclass validation** - Type-safe configs without boilerplate
     - **Dual reference system** - `@` for resolved values, `%` for raw YAML

docs/user-guide/advanced.md

@@ -221,8 +221,6 @@ config.update({"~plugins": [0, 2]})  # Remove list items
 config.update({"~dataloaders": ["train", "test"]})  # Remove dict keys
 ```
 
-**Note:** The `~` directive is idempotent - it doesn't error if the key doesn't exist, enabling reusable configs.
-
 ### Programmatic Updates
 
 Apply operators programmatically:

docs/user-guide/cli.md

@@ -111,7 +111,7 @@ Three operators for fine-grained control:
 |----------|--------|----------|---------|
 | **Compose** (default) | `key=value` | Merges dicts, extends lists | `model::lr=0.001` |
 | **Replace** | `=key=value` | Completely replaces value | `=model={'_target_': 'ResNet'}` |
-| **Delete** | `~key` | Removes key (idempotent) | `~debug` |
+| **Delete** | `~key` | Removes key (errors if missing) | `~debug` |
 
 !!! info "Type Inference"
     Values are automatically typed using `ast.literal_eval()`:

docs/user-guide/operators.md

@@ -126,11 +126,14 @@ Remove keys or list items with `~key`:
 ### Delete Entire Keys
 
 ```yaml
-# Remove keys (idempotent - no error if missing!)
+# Remove keys explicitly
 ~old_param: null
 ~debug_settings: null
 ```
 
+!!! warning "Key Must Exist"
+    The delete operator will raise an error if the key doesn't exist. This helps catch typos and configuration mistakes.
+
 ### Delete Dict Keys
 
 Use path notation for nested keys:
@@ -214,28 +217,6 @@ dataloaders:
 
     **Why?** Path notation is designed for dict keys, not list indices. The batch syntax handles index normalization and processes deletions correctly (high to low order).
 
-### Idempotent Delete
-
-Delete operations don't error if the key doesn't exist:
-
-```yaml
-# production.yaml - Remove debug settings if they exist
-~debug_mode: null
-~dev_logger: null
-~test_data: null
-# No errors if these don't exist!
-```
-
-This enables **reusable configs** that work with multiple bases:
-
-```yaml
-# production.yaml works with ANY base config
-~debug_settings: null
-~verbose_logging: null
-database:
-  pool_size: 100
-```
-
 ## Combining Operators
 
 Mix composition, replace, and delete:
@@ -298,7 +279,7 @@ config.update({"model": {"hidden_size": 1024}})
 # Replace explicitly
 config.update({"=optimizer": {"type": "sgd", "lr": 0.1}})
 
-# Delete keys (idempotent)
+# Delete keys
 config.update({
     "~training::old_param": None,
     "~model::dropout": None
@@ -454,17 +435,40 @@ model:
 
 ### Write Reusable Configs
 
-Use idempotent delete for portable configs:
+!!! warning "Delete Requires Key Existence"
+    The delete operator (`~`) is **strict** - it raises an error if the key doesn't exist. This helps catch typos and configuration mistakes.
 
+When writing configs that should work with different base configurations, you have a few options:
+
+**Option 1: Document required keys**
 ```yaml
-# production.yaml - works with ANY base!
-~debug_mode: null        # Remove if exists
-~verbose_logging: null   # No error if missing
+# production.yaml
+# Requires: base config must have debug_mode and verbose_logging
+~debug_mode: null
+~verbose_logging: null
 database:
   pool_size: 100
   ssl: true
 ```
 
+**Option 2: Use composition order**
+```yaml
+# production.yaml - override instead of delete
+debug_mode: false        # Overrides if exists, sets if not
+verbose_logging: false
+database:
+  pool_size: 100
+  ssl: true
+```
+
+**Option 3: Conditional deletion with lists**
+```yaml
+# Delete multiple optional keys - fails only if ALL are missing
+~: [debug_mode, verbose_logging]  # At least one must exist
+database:
+  pool_size: 100
+```
+
 ## Common Mistakes
 
 ### Using `=` When Not Needed
@@ -519,17 +523,17 @@ plugins: [cache]
 |---------|-------|------------|
 | Dict merge default | Yes ✅ | Yes ✅ |
 | List extend default | No ❌ | **Yes** ✅ |
-| Operators in YAML | No ❌ | Yes ✅ (`=`, `~`) |
-| Operator count | 4 (`+`, `++`, `~`) | **2** (`=`, `~`) ✅ |
-| Delete dict keys | No ❌ | Yes ✅ |
-| Delete list items | No ❌ | Yes ✅ |
-| Idempotent delete | N/A | Yes ✅ |
-
-Sparkwheel goes beyond Hydra with:
-- Full composition-first philosophy (dicts **and** lists)
-- Operators directly in YAML files
-- Just 2 simple operators
-- Delete operations for fine-grained control
+| Operators in YAML | CLI-only | **Yes** ✅ (YAML + CLI) |
+| Operator count | 4 (`=`, `+`, `++`, `~`) | **2** (`=`, `~`) ✅ |
+| Delete dict keys | CLI-only (`~foo.bar`) | **Yes** ✅ (YAML + CLI) |
+| Delete list items | No ❌ | **Yes** ✅ (by index) |
+
+Sparkwheel differs from Hydra:
+- **Full composition philosophy**: Both dicts AND lists compose by default
+- **Operators in YAML files**: Not just CLI overrides
+- **Simpler operator set**: Just 2 operators (`=`, `~`) vs 4 (`=`, `+`, `++`, `~`)
+- **List deletion**: Delete items by index with `~plugins: [0, 2]`
+- **Flexible delete**: Use `~` anywhere (YAML, CLI, programmatic)
 
 ## Next Steps
 

docs/user-guide/references.md

@@ -10,76 +10,88 @@ Sparkwheel provides two types of references for linking configuration values:
 | Feature | `@ref` (Resolved) | `%ref` (Raw) | `$expr` (Expression) |
 |---------|-------------------|--------------|----------------------|
 | **Returns** | Final computed value | Raw YAML content | Evaluated expression result |
-| **When processed** | Lazy (`resolve()`) | Eager (`update()`) | Lazy (`resolve()`) |
+| **When processed** | Lazy (`resolve()`) | External: Eager / Local: Lazy | Lazy (`resolve()`) |
 | **Instantiates objects** | ✅ Yes | ❌ No | ✅ Yes (if referenced) |
 | **Evaluates expressions** | ✅ Yes | ❌ No | ✅ Yes |
 | **Use in dataclass validation** | ✅ Yes | ⚠️ Limited | ✅ Yes |
 | **CLI override compatible** | ✅ Yes | ✅ Yes | ❌ No |
 | **Cross-file references** | ✅ Yes | ✅ Yes | ❌ No |
 | **When to use** | Get computed results | Copy config structures | Compute new values |
 
-## Two-Stage Processing Model
+## Two-Phase Processing Model
 
-Sparkwheel processes references at different times to enable safe config composition:
+Sparkwheel processes raw references (`%`) in two phases to support CLI overrides:
 
 !!! abstract "When References Are Processed"
 
-    **Stage 1: Eager Processing (during `update()`)**
+    **Phase 1: Eager Processing (during `update()`)**
 
-    - **Raw References (`%`)** are expanded immediately when configs are merged
-    - Enables safe config composition and pruning workflows
-    - External file references resolved at load time
+    - **External file raw refs (`%file.yaml::key`)** are expanded immediately
+    - External files are frozen—their content won't change based on CLI overrides
+    - Enables copy-then-delete workflows with external files
 
-    **Stage 2: Lazy Processing (during `resolve()`)**
+    **Phase 2: Lazy Processing (during `resolve()`)**
 
+    - **Local raw refs (`%key`)** are expanded after all composition is complete
     - **Resolved References (`@`)** are processed on-demand
     - **Expressions (`$`)** are evaluated when needed
     - **Components (`_target_`)** are instantiated only when requested
-    - Supports complex dependency graphs and deferred instantiation
+    - CLI overrides can affect local `%` refs
 
-**Why two stages?**
+**Why two phases?**
 
-This separation enables powerful workflows like config pruning:
+This design ensures CLI overrides work intuitively with local raw references:
 
 ```yaml
 # base.yaml
-system:
-  lr: 0.001
-  batch_size: 32
+vars:
+  features_path: null  # Default, will be overridden
 
-experiment:
-  model:
-    optimizer:
-      lr: "%system::lr"  # Copies raw value 0.001 eagerly
-
-~system: null  # Delete system section after copying
+# model.yaml
+dataset:
+  path: "%vars::features_path"  # Local ref - sees CLI override
 ```
 
 ```python
 config = Config()
 config.update("base.yaml")
-# % references already expanded during update()
-# ~system deletion applied after expansion
-# Result: experiment::model::optimizer::lr = 0.001 (system deleted safely)
+config.update("model.yaml")
+config.update("vars::features_path=/data/features.npz")  # CLI override
+
+# Local % ref sees the override!
+path = config.resolve("dataset::path")  # "/data/features.npz"
 ```
 
-With `@` references, this would fail because they resolve lazily after deletion.
+!!! tip "External vs Local Raw References"
+
+    | Type | Example | When Expanded | Use Case |
+    |------|---------|---------------|----------|
+    | **External** | `%file.yaml::key` | Eager (update) | Import from frozen files |
+    | **Local** | `%vars::key` | Lazy (resolve) | Reference config values |
+
+    External files are "frozen"—their content is fixed at load time.
+    Local config values may be overridden via CLI, so local refs see the final state.
 
 ## Resolution Flow
 
 !!! abstract "How References Are Resolved"
 
-    **Step 1: Parse Config** → Detect references in YAML
+    **Step 1: Load Configs** → During `update()`
 
-    **Step 2: Determine Type**
+    - Parse YAML files
+    - Expand external `%file.yaml::key` refs immediately
+    - Keep local `%key` refs as strings
 
-    - **`%key`** → Expanded eagerly during `update()` ✅
-    - **`@key`** → Proceed to dependency resolution (lazy)
+    **Step 2: Apply Overrides** → During `update()` calls
 
-    **Step 3: Resolve Dependencies** (for `@` references during `resolve()`)
+    - CLI overrides modify local config values
+    - Local `%` refs still see the string form
 
+    **Step 3: Resolve** → During `resolve()`
+
+    - Expand local `%key` refs (now sees final values)
+    - Resolve `@` dependencies in order
     - Check for circular references → ❌ **Error if found**
-    - Resolve all dependencies first
     - Evaluate expressions and instantiate objects
     - Return final computed value ✅
 
@@ -223,6 +235,8 @@ model_template: "%base.yaml::model"
 
 ### Local Raw References
 
+Local raw references are expanded lazily during `resolve()`, which means CLI overrides can affect them:
+
 ```yaml
 # config.yaml
 defaults:
@@ -237,6 +251,17 @@ api_config:
 backup_defaults: "%defaults"  # Gets the whole defaults dict
 ```
 
+!!! tip "CLI Overrides Work with Local Raw Refs"
+
+    ```python
+    config = Config()
+    config.update("config.yaml")
+    config.update("defaults::timeout=60")  # CLI override
+
+    # Local % ref sees the override!
+    config.resolve("api_config::timeout")  # 60
+    ```
+
 ### Key Distinction
 
 !!! abstract "@ vs % - When to Use Each"

pyproject.toml

@@ -77,7 +77,6 @@ show_traceback = true
 
 allow_redefinition = false
 check_untyped_defs = true
-disallow_any_generics = true
 disallow_incomplete_defs = true
 ignore_missing_imports = true
 implicit_reexport = false

src/sparkwheel/__init__.py

@@ -5,7 +5,6 @@
 """
 
 from .config import Config, parse_overrides
-from .errors import enable_colors
 from .items import Component, Expression, Instantiable, Item
 from .operators import apply_operators, validate_operators
 from .resolver import Resolver
@@ -19,7 +18,7 @@
     EvaluationError,
     FrozenConfigError,
     InstantiationError,
-    SourceLocation,
+    Location,
     TargetNotFoundError,
 )
 
@@ -39,7 +38,6 @@
     "validate_operators",
     "validate",
     "validator",
-    "enable_colors",
     "RESOLVED_REF_KEY",
     "RAW_REF_KEY",
     "ID_SEP_KEY",
@@ -55,5 +53,5 @@
     "EvaluationError",
     "FrozenConfigError",
     "ValidationError",
-    "SourceLocation",
+    "Location",
 ]