feat: Make Delete Operator Strict, Improve Error Messages, Remove Color Formatting, and Make Local Raw References Lazy (#5)

* Remove idempotent terminology and make delete operator error on missing keys

* Refactor location tracking terminology and enhance merge context

Rename SourceLocation -> Location and MetadataRegistry -> LocationRegistry
to better reflect their purpose. Introduce MergeContext dataclass to
consolidate location tracking during config merging, enabling better error
messages with source locations for delete operator failures.

Key changes:
- Rename src/sparkwheel/metadata.py -> src/sparkwheel/locations.py
- Update all references to SourceLocation to use Location
- Update all references to MetadataRegistry to use LocationRegistry
- Add MergeContext class for threading location info through merge operations
- Enhance delete operator error messages with source location context
- Track locations for individual keys during YAML loading

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* Drop formatters

* Improve error message in instantiationerror

* Add more operator tests

* Improve errors with locations and better messages

* Improve error messages and add comprehensive test coverage for preprocessor

  ## Error Message Improvements

  ### Enhanced delete operator error messages (config.py)
  - Show relevant available keys based on context (top-level vs nested)
  - For nested keys like `model::missing`, display parent container's keys
  - Use `parent_key` parameter for better error formatting
  - Extract child key name for clearer nested error messages

  ### Enhanced raw reference error messages (preprocessor.py)
  - Add source file location tracking for all raw reference errors
  - Show specific missing key instead of full path (e.g., "Key 'data' not found" vs "Key 'data' not found at path 'data'")
  - Display available keys at point of failure (up to 10 keys)
  - Consistent error formatting for dict keys, list indices, and type errors
  - Use custom exceptions (ConfigKeyError, CircularReferenceError) for proper newline formatting
  - Thread LocationRegistry through external file loads for complete error context

  ## Test Coverage

  ### Preprocessor tests (test_preprocessor.py)
  - Added 14 new tests covering:
    - Missing keys (first level and nested)
    - Invalid list indices (first level and nested)
    - Type errors with proper context
    - Raw reference errors with location tracking
    - External file references (success and failure)
    - Circular reference detection with locations
    - Available keys truncation
    - Raw reference expansion scenarios
  - Total: 19 tests, all passing

  ### Config tests (test_config.py)
  - test_delete_nonexistent_top_level_key_shows_available_keys
  - test_delete_nonexistent_nested_key_shows_parent_keys
  - test_delete_nested_key_when_parent_doesnt_exist
  - test_update_from_file_with_nested_paths_merges_locations

  ### Operator tests (test_operators.py)
  - Fixed contradictory comments in test_validate_skips_dict_under_remove

  ## Type Safety

  ### Type annotations (preprocessor.py)
  - Added TYPE_CHECKING import block to avoid circular imports
  - Properly typed `locations` parameter as `Optional["LocationRegistry"]` in:
    - process_raw_refs()
    - _process_raw_refs_recursive()
    - _expand_raw_ref()
  - Removed all `# type: ignore` comments

  ## Documentation

  ### Updated operators.md
  - Removed misleading "idempotent delete" section
  - Clarified delete operator is strict (raises on missing keys)
  - Added three approaches for writing portable/reusable configs:
    1. Document required keys
    2. Use composition/override instead of delete
    3. Conditional deletion with lists
  - Updated examples to reflect strict delete semantics

  ## Results
  - 633 tests passing (1 skipped)
  - Zero regressions
  - Improved error messages show exact file:line and relevant suggestions
  - Complete type safety for location tracking

* Extract get_by_id to path_utils and improve error consistency

  - Move _get_by_id from Preprocessor to standalone get_by_id function in path_utils
  - Config._get_by_id now delegates to the shared get_by_id function
  - Change ValueError to TypeError when indexing non-dict/list values
  - Improve InstantiationError handling to preserve source location and suggestion
  - Move related tests from test_preprocessor.py to test_path_utils.py

* Add a test for instantiation error

* Make local raw references lazy to support CLI overrides

Local % refs (%key) are now expanded during resolve() instead of update(),
allowing CLI overrides to affect values used by local raw references.
External file refs (%file.yaml::key) remain eager since external files are frozen.

This fixes the surprising behavior where CLI overrides were silently ignored
for values referenced by local % refs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

* Update the test

* Update README

---------

Co-authored-by: Claude <[email protected]>

Author: ibro45

.github/workflows/release.yml

@@ -0,0 +1,36 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - '*'
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    name: Create GitHub Release
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    if: startsWith(github.ref, 'refs/tags')
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Verify tag is on main branch
+        run: |
+          git fetch origin main
+          if ! git merge-base --is-ancestor ${{ github.sha }} origin/main; then
+            echo "Error: Tag is not on the main branch"
+            exit 1
+          fi
+
+      - name: Create Release
+        uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true
+          draft: false
+          prerelease: false

README.md

@@ -10,103 +10,56 @@
   <a href="https://github.com/project-lighter/sparkwheel/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"></a>
   <a href="https://project-lighter.github.io/sparkwheel"><img alt="Documentation" src="https://img.shields.io/badge/docs-latest-olive"></a>
 </p>
-<br/>
 
-<p align="center">⚙️ YAML configuration meets Python 🐍</p>
+<h3 align="center">YAML configuration meets Python</h3>
 <p align="center">Define Python objects in YAML. Reference, compose, and instantiate them effortlessly.</p>
 <br/>
 
-## What is Sparkwheel?
+## Quick Start
 
-Stop hardcoding parameters. Define complex Python objects in clean YAML files, compose them naturally, and instantiate with one line.
+```bash
+pip install sparkwheel
+```
 
 ```yaml
 # config.yaml
+dataset:
+  num_classes: 10
+  batch_size: 32
+
 model:
   _target_: torch.nn.Linear
   in_features: 784
-  out_features: "%dataset::num_classes"  # Reference other values
+  out_features: "%dataset::num_classes"  # Reference
 
-dataset:
-  num_classes: 10
+training:
+  steps_per_epoch: "$10000 // @dataset::batch_size"  # Expression
 ```
 
 ```python
 from sparkwheel import Config
 
 config = Config()
 config.update("config.yaml")
-model = config.resolve("model")  # Actual torch.nn.Linear(784, 10) instance!
-```
-
-## Key Features
-
-- **Declarative Object Creation** - Instantiate any Python class from YAML with `_target_`
-- **Smart References** - `@` for resolved values, `%` for raw YAML
-- **Composition by Default** - Configs merge naturally (dicts merge, lists extend)
-- **Explicit Operators** - `=` to replace, `~` to delete when needed
-- **Python Expressions** - Compute values dynamically with `$` prefix
-- **Schema Validation** - Type-check configs with Python dataclasses
-- **CLI Overrides** - Override any value from command line
-
-## Installation
 
-```bash
-pip install sparkwheel
+model = config.resolve("model")  # Actual torch.nn.Linear(784, 10)
 ```
 
-**[→ Get Started in 5 Minutes](https://project-lighter.github.io/sparkwheel/getting-started/quickstart/)**
+## Features
 
-## Coming from Hydra/OmegaConf?
-
-Sparkwheel builds on similar ideas but adds powerful features:
-
-| Feature | Hydra/OmegaConf | Sparkwheel |
-|---------|-----------------|------------|
-| Config composition | Explicit (`+`, `++`) | **By default** (dicts merge, lists extend) |
-| Replace semantics | Default | Explicit with `=` operator |
-| Delete keys | Not idempotent | Idempotent `~` operator |
-| References | OmegaConf interpolation | `@` (resolved) + `%` (raw YAML) |
-| Python expressions | Limited | Full Python with `$` |
-| Schema validation | Structured Configs | Python dataclasses |
-| List extension | Lists replace | **Lists extend by default** |
-
-**Composition by default** means configs merge naturally without operators:
-```yaml
-# base.yaml
-model:
-  hidden_size: 256
-  dropout: 0.1
-
-# experiment.yaml
-model:
-  hidden_size: 512  # Override
-  # dropout inherited
-```
-
-## Documentation
+- **Declarative Objects** - Instantiate any Python class with `_target_`
+- **Smart References** - `@` for resolved values, `%` for raw YAML
+- **Composition by Default** - Dicts merge, lists extend automatically
+- **Explicit Control** - `=` to replace, `~` to delete
+- **Python Expressions** - Dynamic values with `$`
+- **Schema Validation** - Type-check with dataclasses
 
-- [Full Documentation](https://project-lighter.github.io/sparkwheel/)
-- [Quick Start Guide](https://project-lighter.github.io/sparkwheel/getting-started/quickstart/)
-- [Core Concepts](https://project-lighter.github.io/sparkwheel/user-guide/basics/)
-- [API Reference](https://project-lighter.github.io/sparkwheel/reference/)
+**[Get Started](https://project-lighter.github.io/sparkwheel/getting-started/quickstart/)** · **[Documentation](https://project-lighter.github.io/sparkwheel/)** · **[Quick Reference](https://project-lighter.github.io/sparkwheel/user-guide/quick-reference/)**
 
 ## Community
 
-- [Discord Server](https://discord.gg/zJcnp6KrUp) - Chat with the community
-- [YouTube Channel](https://www.youtube.com/channel/UCef1oTpv2QEBrD2pZtrdk1Q) - Tutorials and demos
-- [GitHub Issues](https://github.com/project-lighter/sparkwheel/issues) - Bug reports and feature requests
-
-## Contributing
-
-We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.
+- [Discord](https://discord.gg/zJcnp6KrUp) · [YouTube](https://www.youtube.com/channel/UCef1oTpv2QEBrD2pZtrdk1Q) · [Issues](https://github.com/project-lighter/sparkwheel/issues)
 
 ## About
 
-Sparkwheel is a hard fork of [MONAI Bundle](https://github.com/Project-MONAI/MONAI/tree/dev/monai/bundle)'s configuration system, refined and expanded for general-purpose use. We're deeply grateful to the MONAI team for their excellent foundation.
-
-Sparkwheel powers [Lighter](https://project-lighter.github.io/lighter/), our configuration-driven deep learning framework built on PyTorch Lightning.
-
-## License
-
-Apache License 2.0 - See [LICENSE](LICENSE) for details.
+Sparkwheel is a hard fork of [MONAI Bundle](https://github.com/Project-MONAI/MONAI/tree/dev/monai/bundle)'s config system, with the goal of making a more general-purpose configuration library for Python projects. It combines the best of MONAI Bundle and [Hydra](http://hydra.cc/)/[OmegaComf](https://omegaconf.readthedocs.io/), while introducing new features and improvements not found in either.

docs/index.md

@@ -199,7 +199,7 @@ Sparkwheel has two types of references with distinct purposes:
     - **Composition-by-default** - Configs merge/extend naturally, no operators needed for common case
     - **List extension** - Lists extend by default (unique vs Hydra!)
     - **`=` replace operator** - Explicit control when you need replacement
-    - **`~` delete operator** - Remove inherited keys cleanly (idempotent!)
+    - **`~` delete operator** - Remove inherited keys explicitly
     - **Python expressions with `$`** - Compute values dynamically
     - **Dataclass validation** - Type-safe configs without boilerplate
     - **Dual reference system** - `@` for resolved values, `%` for raw YAML

docs/user-guide/advanced.md

@@ -221,8 +221,6 @@ config.update({"~plugins": [0, 2]})  # Remove list items
 config.update({"~dataloaders": ["train", "test"]})  # Remove dict keys
 ```
 
-**Note:** The `~` directive is idempotent - it doesn't error if the key doesn't exist, enabling reusable configs.
-
 ### Programmatic Updates
 
 Apply operators programmatically:

docs/user-guide/cli.md

@@ -111,7 +111,7 @@ Three operators for fine-grained control:
 |----------|--------|----------|---------|
 | **Compose** (default) | `key=value` | Merges dicts, extends lists | `model::lr=0.001` |
 | **Replace** | `=key=value` | Completely replaces value | `=model={'_target_': 'ResNet'}` |
-| **Delete** | `~key` | Removes key (idempotent) | `~debug` |
+| **Delete** | `~key` | Removes key (errors if missing) | `~debug` |
 
 !!! info "Type Inference"
     Values are automatically typed using `ast.literal_eval()`:

docs/user-guide/operators.md

@@ -126,11 +126,14 @@ Remove keys or list items with `~key`:
 ### Delete Entire Keys
 
 ```yaml
-# Remove keys (idempotent - no error if missing!)
+# Remove keys explicitly
 ~old_param: null
 ~debug_settings: null
 ```
 
+!!! warning "Key Must Exist"
+    The delete operator will raise an error if the key doesn't exist. This helps catch typos and configuration mistakes.
+
 ### Delete Dict Keys
 
 Use path notation for nested keys:
@@ -214,28 +217,6 @@ dataloaders:
 
     **Why?** Path notation is designed for dict keys, not list indices. The batch syntax handles index normalization and processes deletions correctly (high to low order).
 
-### Idempotent Delete
-
-Delete operations don't error if the key doesn't exist:
-
-```yaml
-# production.yaml - Remove debug settings if they exist
-~debug_mode: null
-~dev_logger: null
-~test_data: null
-# No errors if these don't exist!
-```
-
-This enables **reusable configs** that work with multiple bases:
-
-```yaml
-# production.yaml works with ANY base config
-~debug_settings: null
-~verbose_logging: null
-database:
-  pool_size: 100
-```
-
 ## Combining Operators
 
 Mix composition, replace, and delete:
@@ -298,7 +279,7 @@ config.update({"model": {"hidden_size": 1024}})
 # Replace explicitly
 config.update({"=optimizer": {"type": "sgd", "lr": 0.1}})
 
-# Delete keys (idempotent)
+# Delete keys
 config.update({
     "~training::old_param": None,
     "~model::dropout": None
@@ -454,17 +435,40 @@ model:
 
 ### Write Reusable Configs
 
-Use idempotent delete for portable configs:
+!!! warning "Delete Requires Key Existence"
+    The delete operator (`~`) is **strict** - it raises an error if the key doesn't exist. This helps catch typos and configuration mistakes.
 
+When writing configs that should work with different base configurations, you have a few options:
+
+**Option 1: Document required keys**
 ```yaml
-# production.yaml - works with ANY base!
-~debug_mode: null        # Remove if exists
-~verbose_logging: null   # No error if missing
+# production.yaml
+# Requires: base config must have debug_mode and verbose_logging
+~debug_mode: null
+~verbose_logging: null
 database:
   pool_size: 100
   ssl: true
 ```
 
+**Option 2: Use composition order**
+```yaml
+# production.yaml - override instead of delete
+debug_mode: false        # Overrides if exists, sets if not
+verbose_logging: false
+database:
+  pool_size: 100
+  ssl: true
+```
+
+**Option 3: Conditional deletion with lists**
+```yaml
+# Delete multiple optional keys - fails only if ALL are missing
+~: [debug_mode, verbose_logging]  # At least one must exist
+database:
+  pool_size: 100
+```
+
 ## Common Mistakes
 
 ### Using `=` When Not Needed
@@ -519,17 +523,17 @@ plugins: [cache]
 |---------|-------|------------|
 | Dict merge default | Yes ✅ | Yes ✅ |
 | List extend default | No ❌ | **Yes** ✅ |
-| Operators in YAML | No ❌ | Yes ✅ (`=`, `~`) |
-| Operator count | 4 (`+`, `++`, `~`) | **2** (`=`, `~`) ✅ |
-| Delete dict keys | No ❌ | Yes ✅ |
-| Delete list items | No ❌ | Yes ✅ |
-| Idempotent delete | N/A | Yes ✅ |
-
-Sparkwheel goes beyond Hydra with:
-- Full composition-first philosophy (dicts **and** lists)
-- Operators directly in YAML files
-- Just 2 simple operators
-- Delete operations for fine-grained control
+| Operators in YAML | CLI-only | **Yes** ✅ (YAML + CLI) |
+| Operator count | 4 (`=`, `+`, `++`, `~`) | **2** (`=`, `~`) ✅ |
+| Delete dict keys | CLI-only (`~foo.bar`) | **Yes** ✅ (YAML + CLI) |
+| Delete list items | No ❌ | **Yes** ✅ (by index) |
+
+Sparkwheel differs from Hydra:
+- **Full composition philosophy**: Both dicts AND lists compose by default
+- **Operators in YAML files**: Not just CLI overrides
+- **Simpler operator set**: Just 2 operators (`=`, `~`) vs 4 (`=`, `+`, `++`, `~`)
+- **List deletion**: Delete items by index with `~plugins: [0, 2]`
+- **Flexible delete**: Use `~` anywhere (YAML, CLI, programmatic)
 
 ## Next Steps