Skip to content

Evaluate promoting ExtractedAnalysisStore beyond tests #204

@speediedan

Description

@speediedan

Summary

We introduced ExtractedAnalysisStore in tests/analysis_resource_utils.py to let heavy analysis tests copy only a subset of AnalysisStore fields while still preserving by_latent_model(...) behavior for helpers like compute_correct().

This issue is to evaluate whether that pattern should remain a test-only utility or be promoted into a more general feature or shared package utility.

Current use case

  • Low-memory analysis parity tests need to avoid deep-copying full AnalysisStore payloads.
  • Some downstream helpers still expect an AnalysisStore-like object with by_latent_model(...).
  • ExtractedAnalysisStore currently solves that gap inside tests/analysis_resource_utils.py.

Pros of promotion

  • Reduces duplicated lightweight store wrappers in future tests or debugging helpers.
  • Encodes a clear partial-view pattern for AnalysisStore-shaped data.
  • Could be useful outside tests for debugging, profiling, notebook exploration, or selective serialization.
  • Creates one canonical implementation of the by_latent_model(...) subset behavior rather than duplicating it ad hoc.

Cons of promotion

  • Risks turning a test-specific memory workaround into public surface area too early.
  • Could blur the boundary between full AnalysisStore semantics and partial extracted views.
  • May encourage application code to rely on incomplete AnalysisStore objects where the full store is more correct.
  • Adds maintenance burden if AnalysisStore evolves and partial views must track more behavior.

Questions to answer

  1. Should this remain a test-only utility?
  2. If promoted, should it live near AnalysisStore or as a more generic extracted-view helper?
  3. What minimum protocol should it satisfy beyond by_latent_model(...)?
  4. Should we formalize a small protocol for partial analysis-store views instead of promoting the concrete helper?

Related context

This came out of CI memory-pressure work on SAE backend parity tests, especially the NNsight attr-ablation path on GitHub-hosted runners.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions