Merge upstream main, and read the GOC3 case through PowerIO 0.9 - #3
Open
samtalki wants to merge 2 commits into
Open
Merge upstream main, and read the GOC3 case through PowerIO 0.9#3samtalki wants to merge 2 commits into
samtalki wants to merge 2 commits into
Conversation
…onary fallback The parser backend moves from ExaPowerIO's MATPOWER reader to PowerIO, which reads MATPOWER, PSS/E, PowerWorld and PowerModels JSON through one entry point and returns the same NamedTuple layout the model builders already consume. A `from` keyword reaches every model constructor for the cases where the format cannot be told from the extension. ExaPowerIO stays for `get_path(:pglib)`, the bundled case directory, and as the reference the parser tests compare against. Multi-period loads now come from a `PowerIO.LoadSeries` built off the same parsed network as the power data, so the series rows and `data.bus` align by construction rather than by both parsing the file again. The per-period loads are merged onto the bus rows as `busarray` is built, replacing the in-place `update_load_data` mutation. Two behaviour changes fall out: a load curve no longer scales fixed bus shunts, which it did through `gs`/`bs` before, and supplying only one of `pd`/`qd` raises rather than silently leaving the other half at its base-case value. `sc_parser.jl` stops reaching into the raw `sdd_lookup` and `sdd_ts_lookup` dictionaries. Every field it read has a typed home on PowerIO's device rows: `initial_status.p` is `p_0`, `p_shutdown_ramp_ub` is `p_rd_sd`, `p_startup_ramp_ub` is `p_ru_su`, and the `p_lb` series is `p_min`. The four `(device, t, t_prime)` row arrays behind the startup and shutdown power capability sums never left the file, so the sums are accumulated directly; the commitment table is joined by a uid dictionary rather than a scan per row, and the interval end times are computed once instead of re-summing `dt[1:t]` inside the inner loop. Checked against the pre-deletion code on C3E4N00073D1_scenario_303: p_sdpc and all eight sums are bit identical. The parser tests cover the four formats end to end, the isbits row types the GPU backends require, the horizon and load-argument errors, and the shunt behaviour above. The type stability assertion is marked broken: PowerIO 0.9.0's table accessors return the JSON3 value union and its `T` is a keyword-typed `Type`, so the parse cannot be inferred concretely even though every row is concrete at run time.
The DCOPF loop in runtests only evaluated callbacks, so nothing anywhere checked that dcopf_model reaches the right objective, and test_dcopf_case had no call sites at all. Static AC already carves out one solve for the smallest case and leaves every other case on callbacks; DCOPF now gets the same treatment, calling test_dcopf_case on case3 against solve_opf with DCPPowerModel. The reference is cached per case in dcopf_ref_cache, mirroring static_ref_cache, so the two solving backends do not recompute it. The two model builds move out of the callbacks testset so the Float64 variables stay in scope for the solve, which is how the static AC block is already arranged. Adds 8 assertions on the slices that solve: status, objective, and case3's three generator and three branch flows. EMP_TEST_SELECTION=nothing goes 2583 to 2591, EMP_TEST_SELECTION=cpu stays at 450 because CPU() does not solve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
samtalki
force-pushed
the
feat/powerio-0.9-and-upstream-merge
branch
from
August 19, 2026 18:23
cc1e87e to
a78e0cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets
agent/powerio-parser-backend, the head of upstream PR madsuite-org#50. Merging this updates that PR in place, the way fork PRs #1 and #2 did.Do not merge yet. It needs
PowerIO = "0.9", and PowerIO.jl 0.9.0 is not released. See "Before merging" at the end.1. Catch up with upstream main
Upstream added four commits since this PR's merge base. The blocker was madsuite-org#55, which moved every model builder to the ExaModels macro API. Four files conflicted. Everything else merged on its own, and the merged tree has no remaining calls to the retired
variable/constraint/objective/constraint!functions, because this PR changes parsers and not model builders..github/workflows/ci.ymlcudarunner label and per-backend matrix. Keeps the job-level eager registry preference. One job key covers all four legs.src/ExaModelsPower.jlusing DelimitedFiles.src/scopf.jllengthsby name.test/opf_tests.jlTwo of those need explanation.
using DelimitedFilesis the one line a plain merge gets wrong. Upstream kept the import. This PR removed the dependency along with the lastreaddlmcall. Keeping both fails precompilation.lengthsmust be destructured by name. Upstream used positional destructuring.lengthsis a NamedTuple thatsc_parsermerges more names onto, and PowerIO 0.9 addsKto it, so positional binding would silently bind the wrong values.Two more things the merge left incomplete:
test/powerio_parser_tests.jlran its testset at include time, so after the CI split it would run in all four jobs. It is now a function thatruntestscalls from the serial slice only.user_callbackexample still used the old API and returned(vars, cons). Under ExaModels 0.11ExaCoreis immutable and the macros rebind it, so a callback must return(core, vars, cons)or it drops everything it added. Upstream fixedruntests.jland missed the README.2. Read the GOC3 case through PowerIO 0.9
PowerIO.jl PR eigenergy/PowerIO.jl#86 adds the fields this needs.
sc_parserno longer readsparse_goc3_json's raw dictionaries for case data.parse_sc_datadrops itsdata_jsonargument.Kcomes fromlengths, which also removes five inline recomputations of the same expression.p.beta_uband friends instead ofFloat64(data.sdd_lookup[p.uid]["beta_ub"]). PowerIO types those fields, so a typed comprehension keeps a concrete eltype when the filter selects nothing. The four hand-written empty sentinels and theany(...) ? ... : emptyternaries are gone.producers_firstcomes from PowerIO. This derived it from the lexicographically smallest uid, which matches the uid number only when uids are zero padded.scd.violation_cost.j_shinstead of re-parsing the uid string.What still reads
datadirectly: the twogoc3_add_status_flags!arguments, and the startup/shutdown capability builders. Those read fieldsSddRowalready carries, so they need no PowerIO API and can be refactored after 1.0 freezes the surface.3. Solve one DCOPF case
The upstream merge left DCOPF with no solve anywhere, so nothing checked that
dcopf_modelreaches the right objective, andtest_dcopf_casesat inopf_tests.jlwith no call sites at all. Static AC already carves out one solve for the smallest case and leaves every other case on callbacks. DCOPF now gets the same carve-out, callingtest_dcopf_caseon case3 againstsolve_opfwithDCPPowerModel, with the reference cached per case indcopf_ref_cachethe waystatic_ref_cachealready caches the AC references. The two model builds move out of the callbacks testset so theFloat64variables stay in scope for the solve, matching how the static AC block is arranged.This adds 8 assertions on the slices that solve: status, objective, and case3's three generator and three branch flows.
Verification
goc3_modelonC3E4N00073D1_scenario_303gives the same summary before and after:lengths,nvar/ncon, objective and constraint values atx0, and the type and shape of all 60sc_data_arrayfields. It was captured three times — before the merge, after the merge, and after the PowerIO work.One field set changes on purpose.
c_p,c_q,c_sandc_ewereVector{Int64}, because JSON3 reads an integer-valued number asInt64and this wrapped the raw dictionary value untyped. They areFloat64now with the same values. On a GPU backend the old type mixedInt64intoFloat32arithmetic.Local test slices, all green:
EMP_TEST_SELECTION=goc3— 750 testsEMP_TEST_SELECTION=nothing— 2591 tests, 2583 of them before the DCOPF solve was addedEMP_TEST_SELECTION=cpu— 450 tests, unchanged, becauseCPU()is not a solving backendCI also covers
cuda.Before merging
Project.tomlsetsPowerIO = "0.9". Merge this only after PowerIO.jl 0.9.0 registers in General. Until then the test environment cannot resolve.The previous bound was
PowerIO = "0.7.3", which resolves to>=0.7.3, <0.8.0and so excluded every released 0.8.x. The branch could not install PowerIO at all.🤖 Generated with Claude Code