Loaders emit per-test metadata, compact test_id, raw_units (#508)#516
Merged
Conversation
Central hook in from_raw (covers both loader families): - route loader-parsed orphan attributes (test_ID, channel_index, creator, schedule_file_name) into meta_test_dependent so they persist and surface in Data.tests; orphan attributes stay set for backward compatibility (test_name has no home in the model and remains orphan-only) - stamp the compact test_id grouping key (0) onto raw after the loader's intra-file merging and the continuation folds; tester-assigned ids remain as provenance in the box. Fixes the latent cycle_modes_in_data mismatch (raw carried tester ids while the active record is test_id 0) - record load provenance (uuid, source_kind file/db, source_type, source_uri, raw_file_names, loaded_datetime) in Data._provenance (not persisted in v8; #510); build_active_test_meta merges it into the derived TestMeta record Config-driven loaders stamp instrument raw_units on the returned Data via the new shared internal_settings.merge_raw_units helper (fixes the direct-call gap; CellpyCell._set_raw_units delegates to the same helper). Arbin V2-08: Global_Table Comments -> meta_common.comment (when unset); the full vendor-column mapping including deliberately dropped columns is documented in the arbin_res module docstring. Loader goldens regenerated in the follow-up commit. Closes #508. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ta routed into boxes) Expected diffs only: raw.parquet test_id 1/187 -> 0 (arbin/pec) and new test_id column (maccor/neware/custom, metrics n_columns +1); meta.json meta_test_dependent nulls -> parsed values (arbin channel/creator/test_ID, pec test_ID). raw_units goldens unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Linux mdbtools yields NaN where Windows ODBC yields '' for empty Arbin Global_Table values; route neither into the meta boxes (both mean 'not provided') and only map a real non-empty Comments string. Arbin golden test_ID reverts to null accordingly (identical on both platforms now). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #508 — v2 Phase 2 (epic #402, themes V2-05/V2-06/V2-08). Implements the
approved plan (central hook in
from_raw, not per-loader edits).What loads look like now
After any raw load:
test_ID(tester id),channel_index,creator,schedule_file_nameland inmeta_test_dependent— they persist in v8 files and surface inData.tests. Orphan attributes stay set (back-compat;tests/test_pec.pyunchanged).
test_namehas no model home and stays orphan-only.test_idon raw (V2-05a): the raw column holds the grouping key(0 for single, unmerged), stamped after arbin's intra-file merge and the
continuation folds. Tester ids remain as provenance in the box. Fixes the
latent
cycle_modes_in_datapresent-id mismatch.uuid,source_kind(file/db),source_type(instrument),
source_uri,raw_file_names,loaded_datetimerecorded inData._provenanceand merged into the derivedTestMetarecord(
CORE_ONLY_TESTfields — no legacy home by design). Not persisted in v8(v2: cellpy file format v2 + metadata persistence + release (V2-13/14/15) #510); uuid is new per load until then.
raw_unitsby value (V2-05b): config-driven loaders stamp instrumentunits on the returned
Datavia the sharedinternal_settings.merge_raw_units(fixes the direct-loader-call gap;_set_raw_unitsdelegates to the same helper).Global_Table.Comments→meta_common.comment(whenunset); the full vendor-column mapping — including deliberately dropped
columns — is documented in the arbin_res module docstring ("no silent
drops" by documentation).
Goldens (separate commit, expected diffs only)
raw.parquet: arbin/pec
test_id1/187 → 0; maccor/neware/custom gain thecolumn (metrics
n_columns+1). meta.json:meta_test_dependentnulls →parsed values. raw_units goldens byte-identical. EOL-only rewrites from the
regen script were reverted so the diff shows real changes only.
Verification
tests/test_loader_meta.py: 8 passeduv run pytest -m essential: 99 passeduv run pytest: 635 passed, 62 skipped, 14 xfailed🤖 Generated with Claude Code