-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
area: notebooksJupyter notebooksJupyter notebooksarea: testsTesting codeTesting codemodule: analysisAnalysis functionalityAnalysis functionalitymodule: protocolProtocol.py specificProtocol.py specificmodule: runnersRunner implementationsRunner implementationsmodule: utilsUtility functionsUtility functions
Description
Summary
We introduced ExtractedAnalysisStore in tests/analysis_resource_utils.py to let heavy analysis tests copy only a subset of AnalysisStore fields while still preserving by_latent_model(...) behavior for helpers like compute_correct().
This issue is to evaluate whether that pattern should remain a test-only utility or be promoted into a more general feature or shared package utility.
Current use case
- Low-memory analysis parity tests need to avoid deep-copying full
AnalysisStorepayloads. - Some downstream helpers still expect an
AnalysisStore-like object withby_latent_model(...). ExtractedAnalysisStorecurrently solves that gap insidetests/analysis_resource_utils.py.
Pros of promotion
- Reduces duplicated lightweight store wrappers in future tests or debugging helpers.
- Encodes a clear partial-view pattern for
AnalysisStore-shaped data. - Could be useful outside tests for debugging, profiling, notebook exploration, or selective serialization.
- Creates one canonical implementation of the
by_latent_model(...)subset behavior rather than duplicating it ad hoc.
Cons of promotion
- Risks turning a test-specific memory workaround into public surface area too early.
- Could blur the boundary between full
AnalysisStoresemantics and partial extracted views. - May encourage application code to rely on incomplete
AnalysisStoreobjects where the full store is more correct. - Adds maintenance burden if
AnalysisStoreevolves and partial views must track more behavior.
Questions to answer
- Should this remain a test-only utility?
- If promoted, should it live near
AnalysisStoreor as a more generic extracted-view helper? - What minimum protocol should it satisfy beyond
by_latent_model(...)? - Should we formalize a small protocol for partial analysis-store views instead of promoting the concrete helper?
Related context
This came out of CI memory-pressure work on SAE backend parity tests, especially the NNsight attr-ablation path on GitHub-hosted runners.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area: notebooksJupyter notebooksJupyter notebooksarea: testsTesting codeTesting codemodule: analysisAnalysis functionalityAnalysis functionalitymodule: protocolProtocol.py specificProtocol.py specificmodule: runnersRunner implementationsRunner implementationsmodule: utilsUtility functionsUtility functions