Skip to content

Merge upstream main, and read the GOC3 case through PowerIO 0.9 - #3

Open
samtalki wants to merge 2 commits into
mainfrom
feat/powerio-0.9-and-upstream-merge
Open

Merge upstream main, and read the GOC3 case through PowerIO 0.9#3
samtalki wants to merge 2 commits into
mainfrom
feat/powerio-0.9-and-upstream-merge

Conversation

@samtalki

@samtalki samtalki commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Targets agent/powerio-parser-backend, the head of upstream PR madsuite-org#50. Merging this updates that PR in place, the way fork PRs #1 and #2 did.

Do not merge yet. It needs PowerIO = "0.9", and PowerIO.jl 0.9.0 is not released. See "Before merging" at the end.

1. Catch up with upstream main

Upstream added four commits since this PR's merge base. The blocker was madsuite-org#55, which moved every model builder to the ExaModels macro API. Four files conflicted. Everything else merged on its own, and the merged tree has no remaining calls to the retired variable / constraint / objective / constraint! functions, because this PR changes parsers and not model builders.

File Resolution
.github/workflows/ci.yml Upstream's cuda runner label and per-backend matrix. Keeps the job-level eager registry preference. One job key covers all four legs.
src/ExaModelsPower.jl Upstream's macro imports, and drop using DelimitedFiles.
src/scopf.jl Keep destructuring lengths by name.
test/opf_tests.jl Keep the new parser-boundary assertions and upstream's callback smoke check. Drop the solve upstream removed.

Two of those need explanation.

using DelimitedFiles is the one line a plain merge gets wrong. Upstream kept the import. This PR removed the dependency along with the last readdlm call. Keeping both fails precompilation.

lengths must be destructured by name. Upstream used positional destructuring. lengths is a NamedTuple that sc_parser merges more names onto, and PowerIO 0.9 adds K to it, so positional binding would silently bind the wrong values.

Two more things the merge left incomplete:

  • test/powerio_parser_tests.jl ran its testset at include time, so after the CI split it would run in all four jobs. It is now a function that runtests calls from the serial slice only.
  • The README user_callback example still used the old API and returned (vars, cons). Under ExaModels 0.11 ExaCore is immutable and the macros rebind it, so a callback must return (core, vars, cons) or it drops everything it added. Upstream fixed runtests.jl and missed the README.

2. Read the GOC3 case through PowerIO 0.9

PowerIO.jl PR eigenergy/PowerIO.jl#86 adds the fields this needs. sc_parser no longer reads parse_goc3_json's raw dictionaries for case data.

  • parse_sc_data drops its data_json argument. K comes from lengths, which also removes five inline recomputations of the same expression.
  • The four reactive capability arrays read p.beta_ub and friends instead of Float64(data.sdd_lookup[p.uid]["beta_ub"]). PowerIO types those fields, so a typed comprehension keeps a concrete eltype when the filter selects nothing. The four hand-written empty sentinels and the any(...) ? ... : empty ternaries are gone.
  • producers_first comes from PowerIO. This derived it from the lexicographically smallest uid, which matches the uid number only when uids are zero padded.
  • The violation costs come from scd.violation_cost.
  • The shunt rows use PowerIO's j_sh instead of re-parsing the uid string.

What still reads data directly: the two goc3_add_status_flags! arguments, and the startup/shutdown capability builders. Those read fields SddRow already carries, so they need no PowerIO API and can be refactored after 1.0 freezes the surface.

3. Solve one DCOPF case

The upstream merge left DCOPF with no solve anywhere, so nothing checked that dcopf_model reaches the right objective, and test_dcopf_case sat in opf_tests.jl with no call sites at all. Static AC already carves out one solve for the smallest case and leaves every other case on callbacks. DCOPF now gets the same carve-out, calling test_dcopf_case on case3 against solve_opf with DCPPowerModel, with the reference cached per case in dcopf_ref_cache the way static_ref_cache already caches the AC references. The two model builds move out of the callbacks testset so the Float64 variables stay in scope for the solve, matching how the static AC block is arranged.

This adds 8 assertions on the slices that solve: status, objective, and case3's three generator and three branch flows.

Verification

goc3_model on C3E4N00073D1_scenario_303 gives the same summary before and after: lengths, nvar / ncon, objective and constraint values at x0, and the type and shape of all 60 sc_data_array fields. It was captured three times — before the merge, after the merge, and after the PowerIO work.

One field set changes on purpose. c_p, c_q, c_s and c_e were Vector{Int64}, because JSON3 reads an integer-valued number as Int64 and this wrapped the raw dictionary value untyped. They are Float64 now with the same values. On a GPU backend the old type mixed Int64 into Float32 arithmetic.

Local test slices, all green:

  • EMP_TEST_SELECTION=goc3 — 750 tests
  • EMP_TEST_SELECTION=nothing — 2591 tests, 2583 of them before the DCOPF solve was added
  • EMP_TEST_SELECTION=cpu — 450 tests, unchanged, because CPU() is not a solving backend

CI also covers cuda.

Before merging

Project.toml sets PowerIO = "0.9". Merge this only after PowerIO.jl 0.9.0 registers in General. Until then the test environment cannot resolve.

The previous bound was PowerIO = "0.7.3", which resolves to >=0.7.3, <0.8.0 and so excluded every released 0.8.x. The branch could not install PowerIO at all.

🤖 Generated with Claude Code

…onary fallback

The parser backend moves from ExaPowerIO's MATPOWER reader to PowerIO, which
reads MATPOWER, PSS/E, PowerWorld and PowerModels JSON through one entry point
and returns the same NamedTuple layout the model builders already consume. A
`from` keyword reaches every model constructor for the cases where the format
cannot be told from the extension. ExaPowerIO stays for `get_path(:pglib)`, the
bundled case directory, and as the reference the parser tests compare against.

Multi-period loads now come from a `PowerIO.LoadSeries` built off the same
parsed network as the power data, so the series rows and `data.bus` align by
construction rather than by both parsing the file again. The per-period loads
are merged onto the bus rows as `busarray` is built, replacing the in-place
`update_load_data` mutation. Two behaviour changes fall out: a load curve no
longer scales fixed bus shunts, which it did through `gs`/`bs` before, and
supplying only one of `pd`/`qd` raises rather than silently leaving the other
half at its base-case value.

`sc_parser.jl` stops reaching into the raw `sdd_lookup` and `sdd_ts_lookup`
dictionaries. Every field it read has a typed home on PowerIO's device rows:
`initial_status.p` is `p_0`, `p_shutdown_ramp_ub` is `p_rd_sd`,
`p_startup_ramp_ub` is `p_ru_su`, and the `p_lb` series is `p_min`. The four
`(device, t, t_prime)` row arrays behind the startup and shutdown power
capability sums never left the file, so the sums are accumulated directly; the
commitment table is joined by a uid dictionary rather than a scan per row, and
the interval end times are computed once instead of re-summing `dt[1:t]` inside
the inner loop. Checked against the pre-deletion code on
C3E4N00073D1_scenario_303: p_sdpc and all eight sums are bit identical.

The parser tests cover the four formats end to end, the isbits row types the GPU
backends require, the horizon and load-argument errors, and the shunt behaviour
above. The type stability assertion is marked broken: PowerIO 0.9.0's table
accessors return the JSON3 value union and its `T` is a keyword-typed `Type`, so
the parse cannot be inferred concretely even though every row is concrete at run
time.
@samtalki
samtalki changed the base branch from agent/powerio-parser-backend to main August 18, 2026 18:22
@samtalki
samtalki changed the base branch from main to agent/powerio-parser-backend August 18, 2026 18:22
The DCOPF loop in runtests only evaluated callbacks, so nothing anywhere checked that dcopf_model reaches the right objective, and test_dcopf_case had no call sites at all. Static AC already carves out one solve for the smallest case and leaves every other case on callbacks; DCOPF now gets the same treatment, calling test_dcopf_case on case3 against solve_opf with DCPPowerModel. The reference is cached per case in dcopf_ref_cache, mirroring static_ref_cache, so the two solving backends do not recompute it.

The two model builds move out of the callbacks testset so the Float64 variables stay in scope for the solve, which is how the static AC block is already arranged.

Adds 8 assertions on the slices that solve: status, objective, and case3's three generator and three branch flows. EMP_TEST_SELECTION=nothing goes 2583 to 2591, EMP_TEST_SELECTION=cpu stays at 450 because CPU() does not solve.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@samtalki
samtalki force-pushed the feat/powerio-0.9-and-upstream-merge branch from cc1e87e to a78e0cf Compare August 19, 2026 18:23
@samtalki
samtalki changed the base branch from agent/powerio-parser-backend to main August 19, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant