Restyle N-1 SCOPF on the recipe/args split, add DC, wire into AOT - #59
Open
sshin23 wants to merge 8 commits into
Open
Restyle N-1 SCOPF on the recipe/args split, add DC, wire into AOT#59sshin23 wants to merge 8 commits into
sshin23 wants to merge 8 commits into
Conversation
Add a lightweight N-1 security-constrained AC OPF independent of the GOC3
machinery, in both monolithic and Schur-decomposable forms:
- scopf_model (src/scopf_simple.jl): replicates the base AC OPF across a
scenario dimension (column 1 = base, 2..K+1 = contingencies), supporting
generator and branch outages with bounded corrective generator recourse.
Includes the MATPOWER rateA == 0 ("unlimited") fix. polar and rect forms.
- scopf_twostage_model (src/scopf_twostage.jl): the same problem on a
TwoStageExaCore (base = first stage, each contingency = an EachScenario
block), solved with MadNLP's SchurComplementKKTSystem; returns the Schur
partition tags as a 4th value.
- examples/scopf.jl: CLI-driven driver (case/mode/form/gpu/inertia/...) over
the matching <case>.Ctgs line-contingency list, plus case9/case118 data.
- test/scopf_tests.jl: case9 N-1 agreement test (CPU single vs CPU/GPU
two-stage) on objective and base/scenario dispatch, wired into runtests.jl.
- Pin MadNLP + MadNLPGPU to the schur-design-constraints branch via [sources]
so Pkg.test (local and CI) gets the SchurComplementKKTSystem support; the
registry release rejects it. README/docs touchups.
…nto AOT Fold src/scopf_simple.jl and src/scopf_twostage.jl into a single src/scopf.jl, matching opf.jl / mpopf.jl / goc3.jl. Both models are now *_recipe composed with *_args, so the model solved here and the model ExaModelsC compiles come from one definition. The monolithic SCOPF is the multi-period model with the time axis read as a scenario axis. Naming the arrays barray/busarray/arcarray/genarray/refarray and the rep fields as mpopf.jl names them lets its halves build the SCOPF body unchanged: the ramp becomes the corrective coupling and storage drops out. That removes the branching physics (if form == :polar ... elseif :rect) in favour of the existing dispatch on OPFForm. Merge the per-model constraint selector families. Every selector in opf.jl (c_ref, c_angle, c_bal_p, c_bal_q, ac_flow) now takes a trailing idx..., so one definition serves the static model (empty tail), the multi-period model ((t,)) and the SCOPF ((c,)). The 18-method _mp family in mpopf.jl is deleted; it differed from the static one only by the index tail. The splat resolves at compile time -- the tail's length is in the type -- so the traced expression is identical to the hand-written index. Add the DC formulation. A line outage is a mask, and the two families mask different fields. AC zeroes the admittance coefficients c1..c8; DC cannot, because its flow is built from br_r/br_x and zeroing br_x is 0/0 on the zero-resistance branches real cases contain (3 of case9's 9, 9 of case118's 186). So c_ohms_law_dcopf now takes the susceptance as an argument, dc_susceptance(br) computes it as data, and it rides on each branch row, zeroed for the tripped branch in its own scenario. Because the outage becomes a pure data fact, add_flow_cons_mp!(::DC) is reused by the SCOPF unchanged and there is no SCOPF-specific DC code path. scopf_twostage_model stays AC-only. compile_all gains :scacp, :scacr and :scdcp. K is derived from SCOPF_DEFAULT_CONTINGENCIES rather than exposed as a compile_all keyword: the compiled library calls the argument function with a path and nothing else, so the recipe and the arguments cannot be allowed to disagree on K. scopf_tests() sat outside both the EMP_TEST_SELECTION guard and the backend loop, so all four CI jobs ran it; it is now sliced like everything else. The recipe/eager equivalence is checked for all three formulations, DC getting its own loop because static_forms entries carry a PowerModels type and a voltage test that DC lacks. A DC outage regression checks the mechanism rather than the cost: the outaged line's flow must be zero in its own scenario, armed on both sides so that neither an all-zero solution nor a dead scenario would pass.
…of time
`compile_all` failed outright on four `juliac --trim=safe` verifier errors,
all of which bottom out in one line of a dependency: ExaPowerIO's matpower
parser built `bus_map` from an untyped `[]` literal and then converted it by
its `Vector{Int}` annotation. `convert(Vector{Int64}, ::Vector{Any})` copies
element by element through a dynamic `setindex!`, which the verifier will not
resolve, so every caller compiled ahead of time failed on it.
That is fixed upstream, but the newest registered ExaPowerIO is v0.3.0, which
predates it, and upstream has not bumped its version since — so there is no
release to depend on yet. Pin to the commit, in the same form and for the same
reason as the ExaModels pin already in this file: a SHA rather than
`rev = "main"`, because a branch name is not reproducible under
julia-actions/cache, which restores `~/.julia` from an earlier run and reuses
that clone without re-fetching.
With the pin the verifier is clean and `compile_all` produces a working
`libemp.so` carrying all nine prefixes; the `:scacp` library model matches the
eagerly built reference exactly, same dimensions and equal `obj`/`cons` away
from `x0`.
Comes out when ExaPowerIO tags a release carrying the fix.
`test_aot()` sat behind an `EMP_TEST_AOT` environment variable that no job in `ci.yml` sets, so it had never run there. `compile_all` was red on `main` as a result and nothing said so. It is now the `aot` entry in the `EMP_TEST_SELECTION` matrix, beside `goc3`: a slice appears in the job list and cannot be forgotten the way an opt-in variable can. It is also the cheapest slice in the set. The test additionally required `CNLPMODELS_PATH`, since `compile_all` defaults to `path = "@emp"` and a `@name` path resolves against that variable, which nothing in this repository sets. It now passes a directory of its own, so a plain `Pkg.test()` runs it with no setup.
v0.3.1 carries the `bus_map` element-type fix, so the `[sources]` pin has
nothing left to do and comes out.
`[compat]` moves from "0.3" to "0.3.1" rather than staying where it was. "0.3"
still admits v0.3.0, which builds `bus_map` from an untyped `[]` narrowed by
its annotation; resolving to it makes `compile_all` fail on four
`juliac --trim=safe` verifier errors that all bottom out in the resulting
`convert(Vector{Int64}, ::Vector{Any})`. The floor is a build requirement
rather than a preference, so it belongs in the resolver.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #59 +/- ##
==========================================
+ Coverage 80.63% 83.53% +2.89%
==========================================
Files 8 9 +1
Lines 1477 1652 +175
==========================================
+ Hits 1191 1380 +189
+ Misses 286 272 -14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The DC outage regression solved the SCOPF and read the outaged line's flow out of the solution. That made a formulation check depend on a solve, and the solve is the fragile part: MadNLP reaches a primal-feasible point with the right objective (constraint violation 2.7e-15, objective agreeing to 9 significant figures with the converged desktop value) and then terminates INFEASIBLE_PROBLEM_DETECTED at tol=1e-8 on the CI machine. The active constraint Jacobian is rank deficient by exactly K -- summing the bus balance rows over a scenario cancels every branch flow, so the difference of two scenarios closes against the corrective rows -- and the last mu level takes ten inertia corrections before restoration. Whether a particular SCOPF instance is solvable is a separate question from whether it is formulated correctly, and only the second one is this package's responsibility. The DC model is exactly linear, so "the outaged line carries no flow" is a property of the constraint matrix: it holds exactly when pf[l, c] is pinned to a constant by the equalities alone and that constant is zero. That is a row-space membership test, it is exact rather than tolerance-bound, and it holds at every feasible point rather than at one. Measured on case9, case14 and case5_pjm: residual 5.6e-17..2.8e-16 for the outaged pair against 0.058..0.25 for both controls. The arming is unchanged in intent and now also covers the index arithmetic, which is the one place this reaches into ExaModels' variable layout. Mutating the outage away (bs stays at the true susceptance) fails exactly the two outage assertions; mutating every branch out in every scenario fails both arming assertions and the rank precondition. Runs in 35s where the solve took three minutes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on the N-1 SCOPF work by restyling it onto the package's recipe/args split, adding the DC formulation, wiring the security-constrained models into the ahead-of-time compilation workflow, and removing the duplicated per-model constraint-selector families that this work would otherwise have made a third copy of.
One file, built from the multi-period halves
src/scopf_simple.jl+src/scopf_twostage.jlbecome onesrc/scopf.jl, matchingopf.jl/mpopf.jl/goc3.jl.The monolithic SCOPF is the multi-period model with the time axis read as a scenario axis: naming the arrays
barray/busarray/arcarray/genarray/refarrayand therepfields exactly asmpopf.jlnames them lets its halves build the SCOPF body unchanged — the ramp becomes the corrective coupling and storage drops out. That removes the branching physics (if form == :polar … elseif :rect) in favour of the package's existing dispatch onOPFForm.Both models are now
*_recipe∘*_args, so the model solved here and the modelExaModelsCcompiles come from one definition.One selector family instead of three
Every formulation selector in
opf.jl(c_ref,c_angle,c_bal_p,c_bal_q,ac_flow) now takes a trailingidx..., so one definition serves the static model (empty tail →V.va[i]), the multi-period model ((t,)→V.va[i, t]) and the SCOPF ((c,)). The 18-method_mpfamily inmpopf.jlis deleted; it differed from the static one only by the index tail. The splat resolves at compile time — the tail's length is in the type — so the traced expression is identical to the hand-written index.DC formulation
scopf_modelnow takesDC()alongsidePolar()andRect().A line outage is a mask, and the two families mask different fields. AC zeroes the admittance coefficients
c1..c8; DC cannot, because its flow is built frombr_r/br_xand zeroingbr_xis0/0on the zero-resistance branches real cases contain (3 ofcase9's 9, 9 ofcase118's 186). Soc_ohms_law_dcopfnow takes the susceptance as an argument,dc_susceptance(br)computes it as data, and it rides on each branch row — zeroed for the tripped branch in its own scenario.Because the outage becomes a pure data fact,
add_flow_cons_mp!(::DC)is reused by the SCOPF unchanged: there is no SCOPF-specific DC code path.scopf_twostage_modelstays AC-only.AOT
compile_all(ExaModelsPower)gains:scacp,:scacrand:scdcp(nine models).Kis derived fromSCOPF_DEFAULT_CONTINGENCIESrather than exposed as acompile_allkeyword: the compiled library calls the argument function with a path and nothing else, so the recipe and the arguments cannot be allowed to disagree onK.compile_allwas red before this branch as well, onmainand with no SCOPF sources present: fourjuliac --trim=safeverifier errors that all bottom out in one line of a dependency. ExaPowerIO's matpower parser builtbus_mapfrom an untyped[]and narrowed it by itsVector{Int}annotation, and the resultingconvert(Vector{Int64}, ::Vector{Any})copies element by element through a dynamicsetindex!that the verifier will not resolve.That fix is now released as ExaPowerIO v0.3.1, so this branch depends on the registered version rather than pinning a commit.
[compat]moves from"0.3"to"0.3.1":"0.3"still admits v0.3.0, the exact version that cannot be compiled, so the floor is a build requirement rather than a preference and belongs in the resolver.That break went unnoticed because
test_aot()has never run in CI: it sat behind anEMP_TEST_AOTvariable that no job inci.ymlsets. It is now theaotentry in the test matrix, besidegoc3— a slice cannot be forgotten the way an opt-in variable can, and it is the cheapest slice in the set. The variable is gone.test_aot()also no longer needsCNLPMODELS_PATH:compile_alldefaults topath = "@emp", which resolves against that variable, so the test now passes a directory of its own and a plainPkg.test()runs it with no setup.Test wiring
scopf_tests()sat outside both theEMP_TEST_SELECTIONguard and the backend loop, so all four CI jobs ran it. It is now sliced like everything else.The SCOPF recipe/eager equivalence is checked for all three formulations. DC is not in
static_forms(those entries carry a PowerModels type and a voltage test), so it gets its own loop.A DC outage regression checks the mechanism rather than the cost, and checks it on the formulation rather than on a solve. The DC model is exactly linear, so "the outaged line carries no flow" is a property of the constraint matrix: it holds exactly when
pf[l, c]is pinned to a constant by the equalities alone and that constant is zero. That is a row-space membership test — exact rather than tolerance-bound, and true at every feasible point rather than at one. It is armed on both sides: nothing may pin the same line in the base case, and nothing may pin a different line in the same scenario, so neither a formulation that pinned every flow nor a dead scenario would pass. The arming doubles as a check on the index arithmetic, which is the one place this reaches into ExaModels' variable layout.It previously solved and read the flow out of the solution, which made a formulation check depend on the fragile part. Whether a particular SCOPF instance is solvable is a separate question from whether it is formulated correctly, and only the second is this package's responsibility —
case9DC attol = 1e-8is a case where the two answers differ. MadNLP reaches a primal-feasible point with the right objective (constraint violation 2.7e-15, objective agreeing to nine significant figures with a converged run) and then terminatesINFEASIBLE_PROBLEM_DETECTED: the active constraint Jacobian is rank deficient by exactlyK, because summing the bus balance rows over one scenario cancels every branch flow, so the difference of two scenarios closes against the corrective rows. The lastmulevel takes ten inertia corrections before entering restoration. The replacement runs in 35s where the solve took three minutes.Test plan
Run locally on Julia 1.12.6, two NVIDIA GPUs (RTX 3080, RTX 2080 Ti).
EMP_TEST_SELECTION=nothing— 746 passed, 0 failed, 0 errored, 13m26.9s.The ten that would previously have made this 756 are the
aotslicebelow; the compile no longer runs in this one. Wall times here are off a
shared desktop with unmeasured concurrent load, so they are not comparable
run to run.
EMP_TEST_SELECTION=aot— 10 passed, 0 failed, 2m18s, withCNLPMODELS_PATHunset, so the environment is the one CI will have. Ten isexactly
test_aot()'s assertion count, so the slice selects that test andnothing else. Re-run after the pin came out, resolving ExaPowerIO v0.3.1
from the registry rather than from a git checkout — 10 passed, 2m10s.
Before the fix was available it errored inside
compile_allbefore any ofthe ten ran. Both timings are against a warm local depot; a fresh runner
pays precompilation on top.
EMP_TEST_SELECTION=cpu— 450 passed, 0 failedEMP_TEST_SELECTION=cuda— 327 passed, 0 failedseparately because a clean
@testsetprints no per-test breakdown, so theslice being green is not by itself evidence that this path executed.
(the combination that produced the original failure) — 11/11 green, and
each half killed by the mutant aimed at it. Leaving the susceptance at its
true value under outage fails exactly the two outage assertions (residual
0.090 and 0.095, pinned value 0.382 and 0.581) and leaves both arming
assertions green; outaging every branch in every scenario fails both
arming assertions (1.2e-16 … 4.4e-16 against a
> 1e-3bar) and the rankprecondition. Row-space separation holds on
case9,case14_ieeeandcase5_pjm: 5.6e-17 … 2.8e-16 for the outaged pair against 0.058 … 0.25for both controls.
compile_allend-to-end underjuliac --trim=safe— verifier clean,libemp.soproduced with all nine prefixes present, and the:scacplibrary model matches the eagerly built reference: same dimensions
(81, 90), equalobjandconsat a point away fromx0.