Model recipes and ExaModelsC

[sshin23]

An ExaCore is normally built from data you already have, so the model and its
data are finished together and cannot be separated afterwards. This adds a way
to write one against placeholders and supply the data later:

core, N, x0 = ExaCore(concrete = Val(true), nargs = Val(2))
@add_var(core, x, N; start = x0)
@add_obj(core, 100 * (x[i+1] - x[i]^2)^2 + (x[i] - 1)^2 for i = 1:N-1)

ExaModel(core, 10, fill(-1.2, 10))     # one value per placeholder, in order
ExaModel(core, 50, fill(-1.2, 50))     # the same core, again

Such a core is called a recipe. It is not a new type — it is an ordinary
ExaCore holding symbolic expressions in some of its slots instead of numbers.

Why

The reason is ahead-of-time compilation. It needs the structure to become
code while the data stays a run-time input, and juliac --trim=safe requires
the whole call graph to resolve statically. Which modelling constructs survive
that is not something a model author can see while writing; the usual way to
find out is to compile, read a trimming error naming an inferred type somewhere
in the machinery, and guess.

Writing a model against placeholders is the separation compilation requires.
A recipe that builds and instantiates is on the compilation pathway, and the
author never reasons about trimming. One core being reusable at any size is a
consequence, not the motivation.

API

ExaCore(nargs = Val(N)) returns the core followed by N placeholders;
ExaModel(core, args...) consumes one concrete value per placeholder, in
order. nargs = Val(0) is the default and returns the core alone, so nothing
changes for models that do not use this. ntuple over a Val keeps the arity
in the type, so how many values come back is known statically and destructuring
stays inferable.

A placeholder is used as the value it stands for — N is the number of
variables, not a namespace. Sizes, bounds, starting points, the collections
generators iterate, and loop-invariant scalars inside expressions (h * x[i]
with h = 1/(N+1)) can all be deferred. Anything requiring arbitrary Julia — a
comprehension over a deferred size, a random draw, a parsed table — is computed
by the caller and passed as one of the arguments.

Broadcasting over a placeholder is refused with a message saying to compute the
array in the function that builds the arguments. The explicit refusal matters:
Base's broadcastable fallback would otherwise wrap the node in a collect
and fail later as something unrecognisable.

Implementation

ArgSource{K} mirrors DataSource's encoding — a sentinel whose field access
carries the path in a type parameter — but the two are different ideas.
DataSource is the parameterization of an algebraic pattern: the tree is
complete, and the data point is bound afresh each kernel iteration.
ArgSource is a placeholder, substituted once and gone; nothing evaluates it.

To carry deferred values, ExaCore's counters and vectors, SIMDFunction's
offsets, and the lengths of Variable/Parameter/Expression each take a type
parameter. All resolve to Int and the backend's vector type exactly as before,
and the new parameters are appended so existing partial parameterizations still
match. T is now passed explicitly at construction: it used to be recovered
from the VT <: AbstractVector{T} bound, and is still taken from the realised
x0, so backends that promote (Metal, Float64 to Float32) keep their element
type.

The deferral choice is made by dispatch on a Val rather than a predicate over
values, keeping the build path inferable — --trim=safe needs the call graph
resolved, and the AOT test leg passes unchanged.

Because instantiate is the identity on types it has no method for, a field it
cannot walk would pass through with placeholders intact and produce a model
that looks built. ExaModel(core, args...) therefore asks the instantiated
core's type whether any placeholder survives, and reports it. The check is
selected by dispatch on check::Val so it stays trimmable.

ExaModelsC

A subdirectory package compiling a recipe into a shared library through JuliaC,
exposing it over the plain C interface consumed by
CNLPModels.jl and
cnlpmodels:

compile_library("rosenrock", core, 1000)

The example values are given exactly as they would be to ExaModel(core, ...)
— compiling and instantiating a recipe should not be spelled two different
ways — and a schema is derived from their types, one field per placeholder.
The core travels as data and is deserialized at the generated package's
precompile time, so no model-building code enters the compiled call graph. The
example fixes the types trimming needs while the size stays a runtime value. A bare name for the output resolves against CNLPMODELS_PATH, in a
layout both consumers already look for. Both consumers then take their
arguments the same way — CNLPModel("rosenrock", 1000) in Julia,
cnlpmodels.CModel("rosenrock", 1000) in Python — so one spelling runs from
the recipe through the compiler to the model.

bundle = true is the default and carries a privatized copy of the Julia
runtime. That is not merely about self-containment: a library sharing the
host's libjulia aborts on its first call from Julia, because the @ccallable
preamble adopts the calling thread and jl_init_threadtls guards adoption with
if (jl_get_pgcstack() != NULL) abort();. jl_get_pgcstack reads that
runtime's thread-local storage, so a privatized runtime — a distinct one whose
pgcstack is NULL — adopts legitimately. Linking against the installed Julia
and bundling without privatizing both reproduce the abort. bundle = false
gives a single ~2 MB library for Python and C callers, where the calling thread
genuinely is foreign.

Testing

• 151 unit tests for the argument layer, including type-stability assertions on
  both the deterministic and the symbolic build path.
• The compiled library is checked from both consumers against an in-Julia
  ExaModel — metadata, every callback, both sparsity structures, and the
  Ipopt solution. Compiled at N = 4 and exercised at 6, 11, 20 and 25, so a
  size baked in rather than deferred would fail.
• A CI job runs ExaModelsC on self-hosted Linux and macOS and on
  windows-latest, mirroring where the AOT test runs; not on GPU runners.
• Full suite green including the 44-test juliac AOT leg.
• A Literate documentation page whose examples execute at docs build.

Not included

• Structured instantiation for ExaModelsC (the schema + builder ABI, for
  data-defined models such as OPF). An example argument it cannot carry is
  refused with a reason rather than compiled into a library that would fail to
  load.
• add_expr recipes: Expression stores the generator closure, which does not
  serialize, so such a core cannot be compiled.
• Each bundle carries its own runtime. JuliaC salts privatized sonames randomly
  per build; a stable salt would let many models share one copy.
• ~50 MB of every bundle is OpenBLAS and its Fortran dependencies, pulled in by
  OpenBLAS_jll.__init__ because NLPModels depends on LinearAlgebra.
  Nothing in the evaluation kernels calls BLAS.

🤖 Generated with Claude Code

[github-actions]

Your PR requires formatting changes to meet the project's style guidelines.

Please run:

julia --project=@runic -e 'using Pkg; Pkg.add("Runic")'
julia --project=@runic -e "using Runic; exit(Runic.main(ARGS))" -- --fix <files>

(or git runic main if you have the git wrapper installed)

Note: the full diff is omitted because it can exceed GitHub Actions input limits.

[sshin23]

@pulsipher fyi. With this PR, ExaModels allows creating a "recipe", which you can compile a *.so file that can be consumed via CNLPModels.jl or cnlpmodels-py. There is no need for a separate code generation, so my hope is that the AOT compatibility becomes more broadly useful (e.g., users via InfiteOpt.jl).

[github-actions]

Benchmark Results