Skip to content

fix(runtime): copy deeply nested terms without native recursion - #25

Merged
mremond merged 1 commit into
mainfrom
fm/task-60e57963
Nov 13, 2026
Merged

mremond merged 1 commit into
mainfrom
fm/task-60e57963

Conversation

@mremond

@mremond mremond commented Nov 13, 2026

Copy link
Copy Markdown
Member

Why

A term nested hundreds of thousands of levels deep, such as an iolist built by prepending, crashed the program with a bus error when it was sent, captured in a spawned fun or stored in ETS, while BEAM handles it.

What changed

  • Deeply nested iolists, tuples, maps and fun chains can be sent to other processes and captured in spawned funs.
  • Such terms can be stored in ETS or persistent_term, read back, and used as keys of a set table.
  • Inserting rows that hold lists, tuples or maps into ETS, and ets:tab2list/1, run faster.

Before and after

Before: A probe on the old version crashed in all 18 of its 200,000-level cases (send, spawn, ETS lookup, tab2list, ETS key, persistent_term; iolist, tuple, map), and ETS crashed from about 5,000 levels.

After: All 36 probe cases, at 1,000 and 200,000 levels, return the full-depth term, and the new fixture passes natively and on OTP 29.

This is a runtime change with nothing on screen; program output shows the result.

Checked

  • Crash probe at 1,000 and 200,000 levels for send, spawn, ETS lookup, tab2list, ETS key and persistent_term, with iolist, tuple and map: 36 of 36 pass; the old version crashed in 18 of 18 deep cases
  • New copy_deep_nesting fixture, ten runs in a row natively and ten on OTP 29: 20 of 20 passed, no retries
  • Benchmarks against the old version in ten alternating runs: ETS insert of nested rows 32% faster and tab2list 54% faster; send, spawn and flat rows within noise
  • Ordered_set key nested 8,000 or more levels: Still crashes in term comparison, as listed in known limits
How it was checked

cargo build --release --locked -p nera-runtime and cargo build --release --locked

cargo clippy --workspace --all-targets --locked -- -D warnings

cargo test --workspace --locked --no-fail-fast: 2,145 passed

python3 scripts/test-allocator-config.py

nera test -j 1 --timeout 30: 480 passed, 2 skipped (need external modules)

Risk

Risk above low: Blast radius, medium — Message, spawn, ETS and persistent_term copies run in every program; Novelty, medium — A new work-stack module beside the existing copiers

Full risk assessment
Dimension Level Why
Blast radius medium Message, spawn, ETS and persistent_term copies run in every program
Behavior low A crash is corrected; copied terms keep the same layout
Undo low Revert the commit; nothing stored or sent changes format
Evidence low Probe, fixture and full suite shown working natively and on OTP 29
Sensitivity low No sign-ins, keys, saved data or releases involved
Novelty medium A new work-stack module beside the existing copiers
Dependencies low No new or updated packages
Not done

Comparing such terms, using one as an ordered_set key, term_to_binary/1, erlang:phash2/1 and iolist_to_binary/1 still recurse and crash at the depths known limits gives.

A term nested hundreds of thousands of levels deep, such as an iolist
built by prepending or a chain of tuples or maps, can now be sent to
another process, captured in a spawned fun, stored in ETS or
persistent_term and read back, as on BEAM. The message copier and the
ETS copies used to recurse on the native stack for list heads and tuple,
map and closure elements, so such a term killed the program with a bus
error: in ETS from about 5,000 levels, when sent or captured from about
14,000.

The message copier and the ETS copies now copy depth first and keep the
slots still to copy on a heap-allocated work stack, as the garbage
collector does since it stopped recursing. The ETS key encoder and the
word count for a stored entry use the same kind of stack. Each work stack
reuses its thread's spare buffer, so copying an ordinary term allocates
nothing extra. The ETS copies also stop building a temporary vector for
each list, tuple or map, which makes inserting nested rows and
ets:tab2list/1 faster.

The copy_deep_nesting fixture sends, captures and stores 200,000-level
terms and looks one up as a key, natively and on BEAM.
@github-actions github-actions Bot added this to the 0.1.1 milestone Nov 13, 2026
@mremond
mremond merged commit 2b5d015 into main Nov 13, 2026
22 checks passed
@mremond
mremond deleted the fm/task-60e57963 branch November 14, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant