Validate circuit-tracer abstraction layer completeness

[speediedan]

Summary

Validate completeness of the circuit-tracer adapter abstraction layer introduced in the NNsight support workstream (S1–S9). The NNsight backend for circuit-tracer is functional and passes all parity tests, but several abstraction-layer concerns need validation before the circuit-tracer integration can be considered production-ready.

Priority: High

These items affect the long-term maintainability and correctness of the circuit-tracer adapter across both TransformerLens and NNsight backends.

Tasks

1. AnalysisStore backend-agnostic validation

Verify that AnalysisStore containers produced by circuit-tracer analysis operations are fully backend-agnostic — i.e., a consumer of analysis results cannot distinguish whether TransformerLens or NNsight was the execution backend. Check all stored tensor keys, metadata fields, and dataset columns.

2. Serialized artifact parity

Confirm that serialized analysis artifacts (saved via AnalysisStore.save() or similar) are byte-identical or structurally equivalent across backends. This includes tensor .pt files, dataset parquet shards, and any YAML/JSON metadata.

3. latent_model_handles lifecycle validation

The latent_model_handles mechanism (used for SAE activation caching during circuit-tracer graph construction) should be validated for proper cleanup across both backends. Ensure handles are properly released after analysis completes and no hooks/traces leak.

4. Cross-backend smoke test

Create a minimal end-to-end smoke test that:

• Runs a circuit-tracer analysis with TransformerLens backend
• Runs the same analysis with NNsight backend
• Compares the resulting graph structures (node counts, edge counts, feature attributions)
• Currently only unit-level parity tests exist (test_adapters_circuit_tracer.py); a higher-level integration test would catch composition-level issues.

5. Protocol gaps documentation

Document any CircuitTracerProtocol or AnalysisOpProtocol methods that are backend-specific or have different semantics between TransformerLens and NNsight. Update docs/adapter_development_guide.md with a "Backend Compatibility Matrix" section.

Context

• Branch: nnsight-support
• Related files:
  • src/interpretune/adapters/circuit_tracer.py — adapter with dual-backend support
  • src/interpretune/analysis/ — analysis operations and AnalysisStore
  • tests/core/test_adapters_circuit_tracer.py — existing unit tests
  • tests/core/test_sae_backend_parity.py — SAE-level backend parity tests (reference pattern)

[speediedan]

CT_BACKEND_REGISTRY Implementation Update

The isinstance() bandaid fix for circuit-tracer backend validation has been replaced with a proper CT_BACKEND_REGISTRY pattern in src/interpretune/adapters/circuit_tracer.py.

What changed

• App code: Added CT_BACKEND_REGISTRY: dict[str, str] mapping backend names to expected class names ("transformerlens" → "TransformerLensReplacementModel", "nnsight" → "NNSightReplacementModel"). Validation in _load_replacement_model() now uses type(replacement_model).__name__ against the registry instead of isinstance(), which avoids the importlib double-loading issue.
• Tests: All 4 isinstance() string comparison hacks in test_adapters_circuit_tracer.py have been reverted to proper isinstance() checks. Tests validate type identity at the Python level, while app code validates by class name (resilient to module double-loading).

Future consideration: backend registry name collision

When circuit-tracer adds new backends or if upstream renames classes, the CT_BACKEND_REGISTRY entries must be kept in sync. If two different backend implementations ever share a class name, the registry would need to include module path (e.g., "circuit_tracer.replacement_models.TransformerLensReplacementModel") rather than just class name. For now, class names are unique across backends so simple name matching suffices.