Skip to content

Restyle N-1 SCOPF on the recipe/args split, add DC, wire into AOT - #59

Open
sshin23 wants to merge 8 commits into
mainfrom
relay/scopf-rebase
Open

Restyle N-1 SCOPF on the recipe/args split, add DC, wire into AOT#59
sshin23 wants to merge 8 commits into
mainfrom
relay/scopf-rebase

Conversation

@sshin23

@sshin23 sshin23 commented Aug 16, 2026

Copy link
Copy Markdown
Member

Summary

Builds on the N-1 SCOPF work by restyling it onto the package's recipe/args split, adding the DC formulation, wiring the security-constrained models into the ahead-of-time compilation workflow, and removing the duplicated per-model constraint-selector families that this work would otherwise have made a third copy of.

One file, built from the multi-period halves

src/scopf_simple.jl + src/scopf_twostage.jl become one src/scopf.jl, matching opf.jl / mpopf.jl / goc3.jl.

The monolithic SCOPF is the multi-period model with the time axis read as a scenario axis: naming the arrays barray/busarray/arcarray/genarray/refarray and the rep fields exactly as mpopf.jl names them lets its halves build the SCOPF body unchanged — the ramp becomes the corrective coupling and storage drops out. That removes the branching physics (if form == :polar … elseif :rect) in favour of the package's existing dispatch on OPFForm.

Both models are now *_recipe*_args, so the model solved here and the model ExaModelsC compiles come from one definition.

One selector family instead of three

Every formulation selector in opf.jl (c_ref, c_angle, c_bal_p, c_bal_q, ac_flow) now takes a trailing idx..., so one definition serves the static model (empty tail → V.va[i]), the multi-period model ((t,)V.va[i, t]) and the SCOPF ((c,)). The 18-method _mp family in mpopf.jl is deleted; it differed from the static one only by the index tail. The splat resolves at compile time — the tail's length is in the type — so the traced expression is identical to the hand-written index.

DC formulation

scopf_model now takes DC() alongside Polar() and Rect().

A line outage is a mask, and the two families mask different fields. AC zeroes the admittance coefficients c1..c8; DC cannot, because its flow is built from br_r/br_x and zeroing br_x is 0/0 on the zero-resistance branches real cases contain (3 of case9's 9, 9 of case118's 186). So c_ohms_law_dcopf now takes the susceptance as an argument, dc_susceptance(br) computes it as data, and it rides on each branch row — zeroed for the tripped branch in its own scenario.

Because the outage becomes a pure data fact, add_flow_cons_mp!(::DC) is reused by the SCOPF unchanged: there is no SCOPF-specific DC code path.

scopf_twostage_model stays AC-only.

AOT

compile_all(ExaModelsPower) gains :scacp, :scacr and :scdcp (nine models). K is derived from SCOPF_DEFAULT_CONTINGENCIES rather than exposed as a compile_all keyword: the compiled library calls the argument function with a path and nothing else, so the recipe and the arguments cannot be allowed to disagree on K.

compile_all was red before this branch as well, on main and with no SCOPF sources present: four juliac --trim=safe verifier errors that all bottom out in one line of a dependency. ExaPowerIO's matpower parser built bus_map from an untyped [] and narrowed it by its Vector{Int} annotation, and the resulting convert(Vector{Int64}, ::Vector{Any}) copies element by element through a dynamic setindex! that the verifier will not resolve.

That fix is now released as ExaPowerIO v0.3.1, so this branch depends on the registered version rather than pinning a commit. [compat] moves from "0.3" to "0.3.1": "0.3" still admits v0.3.0, the exact version that cannot be compiled, so the floor is a build requirement rather than a preference and belongs in the resolver.

That break went unnoticed because test_aot() has never run in CI: it sat behind an EMP_TEST_AOT variable that no job in ci.yml sets. It is now the aot entry in the test matrix, beside goc3 — a slice cannot be forgotten the way an opt-in variable can, and it is the cheapest slice in the set. The variable is gone. test_aot() also no longer needs CNLPMODELS_PATH: compile_all defaults to path = "@emp", which resolves against that variable, so the test now passes a directory of its own and a plain Pkg.test() runs it with no setup.

Test wiring

  • scopf_tests() sat outside both the EMP_TEST_SELECTION guard and the backend loop, so all four CI jobs ran it. It is now sliced like everything else.

  • The SCOPF recipe/eager equivalence is checked for all three formulations. DC is not in static_forms (those entries carry a PowerModels type and a voltage test), so it gets its own loop.

  • A DC outage regression checks the mechanism rather than the cost, and checks it on the formulation rather than on a solve. The DC model is exactly linear, so "the outaged line carries no flow" is a property of the constraint matrix: it holds exactly when pf[l, c] is pinned to a constant by the equalities alone and that constant is zero. That is a row-space membership test — exact rather than tolerance-bound, and true at every feasible point rather than at one. It is armed on both sides: nothing may pin the same line in the base case, and nothing may pin a different line in the same scenario, so neither a formulation that pinned every flow nor a dead scenario would pass. The arming doubles as a check on the index arithmetic, which is the one place this reaches into ExaModels' variable layout.

    It previously solved and read the flow out of the solution, which made a formulation check depend on the fragile part. Whether a particular SCOPF instance is solvable is a separate question from whether it is formulated correctly, and only the second is this package's responsibility — case9 DC at tol = 1e-8 is a case where the two answers differ. MadNLP reaches a primal-feasible point with the right objective (constraint violation 2.7e-15, objective agreeing to nine significant figures with a converged run) and then terminates INFEASIBLE_PROBLEM_DETECTED: the active constraint Jacobian is rank deficient by exactly K, because summing the bus balance rows over one scenario cancels every branch flow, so the difference of two scenarios closes against the corrective rows. The last mu level takes ten inertia corrections before entering restoration. The replacement runs in 35s where the solve took three minutes.

Test plan

Run locally on Julia 1.12.6, two NVIDIA GPUs (RTX 3080, RTX 2080 Ti).

  • EMP_TEST_SELECTION=nothing — 746 passed, 0 failed, 0 errored, 13m26.9s.
    The ten that would previously have made this 756 are the aot slice
    below; the compile no longer runs in this one. Wall times here are off a
    shared desktop with unmeasured concurrent load, so they are not comparable
    run to run.
  • EMP_TEST_SELECTION=aot — 10 passed, 0 failed, 2m18s, with
    CNLPMODELS_PATH unset, so the environment is the one CI will have. Ten is
    exactly test_aot()'s assertion count, so the slice selects that test and
    nothing else. Re-run after the pin came out, resolving ExaPowerIO v0.3.1
    from the registry rather than from a git checkout — 10 passed, 2m10s.
    Before the fix was available it errored inside compile_all before any of
    the ten ran. Both timings are against a warm local depot; a fresh runner
    pays precompilation on top.
  • EMP_TEST_SELECTION=cpu — 450 passed, 0 failed
  • EMP_TEST_SELECTION=cuda — 327 passed, 0 failed
  • GPU SCOPF two-stage, in isolation — 5 passed, 0 failed, 2m54s. Run
    separately because a clean @testset prints no per-test breakdown, so the
    slice being green is not by itself evidence that this path executed.
  • DC outage assertion, mutation tested on the CI machine under Julia 1.12.7
    (the combination that produced the original failure) — 11/11 green, and
    each half killed by the mutant aimed at it. Leaving the susceptance at its
    true value under outage fails exactly the two outage assertions (residual
    0.090 and 0.095, pinned value 0.382 and 0.581) and leaves both arming
    assertions green; outaging every branch in every scenario fails both
    arming assertions (1.2e-16 … 4.4e-16 against a > 1e-3 bar) and the rank
    precondition. Row-space separation holds on case9, case14_ieee and
    case5_pjm: 5.6e-17 … 2.8e-16 for the outaged pair against 0.058 … 0.25
    for both controls.
  • compile_all end-to-end under juliac --trim=safe — verifier clean,
    libemp.so produced with all nine prefixes present, and the :scacp
    library model matches the eagerly built reference: same dimensions
    (81, 90), equal obj and cons at a point away from x0.

michel2323 and others added 4 commits August 15, 2026 18:06
Add a lightweight N-1 security-constrained AC OPF independent of the GOC3
machinery, in both monolithic and Schur-decomposable forms:

- scopf_model (src/scopf_simple.jl): replicates the base AC OPF across a
  scenario dimension (column 1 = base, 2..K+1 = contingencies), supporting
  generator and branch outages with bounded corrective generator recourse.
  Includes the MATPOWER rateA == 0 ("unlimited") fix. polar and rect forms.
- scopf_twostage_model (src/scopf_twostage.jl): the same problem on a
  TwoStageExaCore (base = first stage, each contingency = an EachScenario
  block), solved with MadNLP's SchurComplementKKTSystem; returns the Schur
  partition tags as a 4th value.
- examples/scopf.jl: CLI-driven driver (case/mode/form/gpu/inertia/...) over
  the matching <case>.Ctgs line-contingency list, plus case9/case118 data.
- test/scopf_tests.jl: case9 N-1 agreement test (CPU single vs CPU/GPU
  two-stage) on objective and base/scenario dispatch, wired into runtests.jl.
- Pin MadNLP + MadNLPGPU to the schur-design-constraints branch via [sources]
  so Pkg.test (local and CI) gets the SchurComplementKKTSystem support; the
  registry release rejects it. README/docs touchups.
…nto AOT

Fold src/scopf_simple.jl and src/scopf_twostage.jl into a single src/scopf.jl,
matching opf.jl / mpopf.jl / goc3.jl. Both models are now *_recipe composed with
*_args, so the model solved here and the model ExaModelsC compiles come from one
definition.

The monolithic SCOPF is the multi-period model with the time axis read as a
scenario axis. Naming the arrays barray/busarray/arcarray/genarray/refarray and
the rep fields as mpopf.jl names them lets its halves build the SCOPF body
unchanged: the ramp becomes the corrective coupling and storage drops out. That
removes the branching physics (if form == :polar ... elseif :rect) in favour of
the existing dispatch on OPFForm.

Merge the per-model constraint selector families. Every selector in opf.jl
(c_ref, c_angle, c_bal_p, c_bal_q, ac_flow) now takes a trailing idx..., so one
definition serves the static model (empty tail), the multi-period model ((t,))
and the SCOPF ((c,)). The 18-method _mp family in mpopf.jl is deleted; it
differed from the static one only by the index tail. The splat resolves at
compile time -- the tail's length is in the type -- so the traced expression is
identical to the hand-written index.

Add the DC formulation. A line outage is a mask, and the two families mask
different fields. AC zeroes the admittance coefficients c1..c8; DC cannot,
because its flow is built from br_r/br_x and zeroing br_x is 0/0 on the
zero-resistance branches real cases contain (3 of case9's 9, 9 of case118's
186). So c_ohms_law_dcopf now takes the susceptance as an argument,
dc_susceptance(br) computes it as data, and it rides on each branch row, zeroed
for the tripped branch in its own scenario. Because the outage becomes a pure
data fact, add_flow_cons_mp!(::DC) is reused by the SCOPF unchanged and there is
no SCOPF-specific DC code path. scopf_twostage_model stays AC-only.

compile_all gains :scacp, :scacr and :scdcp. K is derived from
SCOPF_DEFAULT_CONTINGENCIES rather than exposed as a compile_all keyword: the
compiled library calls the argument function with a path and nothing else, so
the recipe and the arguments cannot be allowed to disagree on K.

scopf_tests() sat outside both the EMP_TEST_SELECTION guard and the backend
loop, so all four CI jobs ran it; it is now sliced like everything else. The
recipe/eager equivalence is checked for all three formulations, DC getting its
own loop because static_forms entries carry a PowerModels type and a voltage
test that DC lacks. A DC outage regression checks the mechanism rather than the
cost: the outaged line's flow must be zero in its own scenario, armed on both
sides so that neither an all-zero solution nor a dead scenario would pass.
@sshin23
sshin23 requested a review from michel2323 August 16, 2026 02:45
@sshin23
sshin23 changed the base branch from scopf-schur-twostage to main August 16, 2026 12:33
…of time

`compile_all` failed outright on four `juliac --trim=safe` verifier errors,
all of which bottom out in one line of a dependency: ExaPowerIO's matpower
parser built `bus_map` from an untyped `[]` literal and then converted it by
its `Vector{Int}` annotation. `convert(Vector{Int64}, ::Vector{Any})` copies
element by element through a dynamic `setindex!`, which the verifier will not
resolve, so every caller compiled ahead of time failed on it.

That is fixed upstream, but the newest registered ExaPowerIO is v0.3.0, which
predates it, and upstream has not bumped its version since — so there is no
release to depend on yet. Pin to the commit, in the same form and for the same
reason as the ExaModels pin already in this file: a SHA rather than
`rev = "main"`, because a branch name is not reproducible under
julia-actions/cache, which restores `~/.julia` from an earlier run and reuses
that clone without re-fetching.

With the pin the verifier is clean and `compile_all` produces a working
`libemp.so` carrying all nine prefixes; the `:scacp` library model matches the
eagerly built reference exactly, same dimensions and equal `obj`/`cons` away
from `x0`.

Comes out when ExaPowerIO tags a release carrying the fix.
`test_aot()` sat behind an `EMP_TEST_AOT` environment variable that no job in
`ci.yml` sets, so it had never run there. `compile_all` was red on `main` as a
result and nothing said so. It is now the `aot` entry in the
`EMP_TEST_SELECTION` matrix, beside `goc3`: a slice appears in the job list and
cannot be forgotten the way an opt-in variable can. It is also the cheapest
slice in the set.

The test additionally required `CNLPMODELS_PATH`, since `compile_all` defaults
to `path = "@emp"` and a `@name` path resolves against that variable, which
nothing in this repository sets. It now passes a directory of its own, so a
plain `Pkg.test()` runs it with no setup.
v0.3.1 carries the `bus_map` element-type fix, so the `[sources]` pin has
nothing left to do and comes out.

`[compat]` moves from "0.3" to "0.3.1" rather than staying where it was. "0.3"
still admits v0.3.0, which builds `bus_map` from an untyped `[]` narrowed by
its annotation; resolving to it makes `compile_all` fail on four
`juliac --trim=safe` verifier errors that all bottom out in the resulting
`convert(Vector{Int64}, ::Vector{Any})`. The floor is a build requirement
rather than a preference, so it belongs in the resolver.
@codecov

codecov Bot commented Aug 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.40678% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.53%. Comparing base (5f64c18) to head (ba19ee4).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/scopf.jl 88.29% 24 Missing ⚠️
src/constraint.jl 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #59      +/-   ##
==========================================
+ Coverage   80.63%   83.53%   +2.89%     
==========================================
  Files           8        9       +1     
  Lines        1477     1652     +175     
==========================================
+ Hits         1191     1380     +189     
+ Misses        286      272      -14     
Flag Coverage Δ
aot 21.73% <45.92%> (?)
cpu 21.34% <12.34%> (-3.93%) ⬇️
cuda 32.52% <89.69%> (+7.25%) ⬆️
goc3 51.93% <0.00%> (-6.41%) ⬇️
nothing 33.89% <90.12%> (+8.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The DC outage regression solved the SCOPF and read the outaged line's flow
out of the solution. That made a formulation check depend on a solve, and
the solve is the fragile part: MadNLP reaches a primal-feasible point with
the right objective (constraint violation 2.7e-15, objective agreeing to 9
significant figures with the converged desktop value) and then terminates
INFEASIBLE_PROBLEM_DETECTED at tol=1e-8 on the CI machine. The active
constraint Jacobian is rank deficient by exactly K -- summing the bus
balance rows over a scenario cancels every branch flow, so the difference of
two scenarios closes against the corrective rows -- and the last mu level
takes ten inertia corrections before restoration. Whether a particular SCOPF
instance is solvable is a separate question from whether it is formulated
correctly, and only the second one is this package's responsibility.

The DC model is exactly linear, so "the outaged line carries no flow" is a
property of the constraint matrix: it holds exactly when pf[l, c] is pinned
to a constant by the equalities alone and that constant is zero. That is a
row-space membership test, it is exact rather than tolerance-bound, and it
holds at every feasible point rather than at one. Measured on case9, case14
and case5_pjm: residual 5.6e-17..2.8e-16 for the outaged pair against
0.058..0.25 for both controls.

The arming is unchanged in intent and now also covers the index arithmetic,
which is the one place this reaches into ExaModels' variable layout. Mutating
the outage away (bs stays at the true susceptance) fails exactly the two
outage assertions; mutating every branch out in every scenario fails both
arming assertions and the rank precondition.

Runs in 35s where the solve took three minutes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants