nera needs a standard library that provides Erlang developers with the
modules they expect (gen_server, json, lists, file, etc.). The key
design question is: what should be implemented in Erlang (compiled by nera)
versus in Rust (linked into the runtime)?
The answer is both, at different layers. This document defines a three-tier architecture that places each piece of functionality at the appropriate level based on its nature, performance requirements, and coupling to the runtime.
┌──────────────────────────────────────────────────────────┐
│ Tier 3 — Erlang Libraries │
│ gen_server.erl, supervisor.erl, proc_lib.erl │
│ Written in Erlang, compiled by nera alongside user code │
│ Depends on: Tier 1 primitives + Tier 2 libraries │
├──────────────────────────────────────────────────────────┤
│ Tier 2 — Rust Extension Libraries │
│ json, re, crypto, base64, file, unicode, http_client │
│ Pure functions: ErlTerm → Rust → ErlTerm │
│ No scheduler/GC coupling. Leverages Rust crate ecosystem │
├──────────────────────────────────────────────────────────┤
│ Tier 1 — Core Runtime Primitives │
│ Scheduler, heap/GC, process management, message passing, │
│ links, monitors, EXIT signals, atom table, process │
│ registry, ETS, timers │
│ Tightly coupled to scheduler internals │
└──────────────────────────────────────────────────────────┘
| Question | Tier 1 (Rust, runtime) | Tier 2 (Rust, extension) | Tier 3 (Erlang) |
|---|---|---|---|
| Does it need scheduler access? | Yes | No | No |
| Does it manipulate process state? | Yes | No | Via message passing |
| Is there a mature Rust crate? | N/A | Yes | N/A |
| Would a pure Erlang impl be 10x+ slower? | N/A | Yes | No |
| Is it naturally expressed as message-passing patterns? | No | No | Yes |
For new stdlib APIs, the default is Tier 3 (Erlang module).
Why this is the default:
- Erlang code naturally cooperates with reduction-based scheduling.
- It avoids long native sections that can delay scheduler fairness.
- For many utility functions, Rust BIF vs Erlang performance is often close enough that maintainability wins.
Promote to a Rust BIF (Tier 1 or Tier 2) only when at least one condition holds:
- The feature needs runtime internals Erlang cannot access (scheduler, process internals, term representation, timers, ETS internals).
- The feature needs semantics that cannot be expressed safely in Erlang (for example, specific atomicity guarantees).
- Benchmarks on representative workloads show a material win that matters in practice (guideline: at least ~2x end-to-end for the targeted operation).
If implemented as a BIF, it must still be scheduler-safe: bounded work per call, no blocking on scheduler threads, and proper reduction accounting (or trap/yield behavior).
These are implemented in Rust inside crates/nera-runtime/ because they
require direct access to scheduler internals, process state, or memory
management.
| Primitive | Location | Purpose |
|---|---|---|
| Scheduler (N:M work-stealing) | scheduler.rs |
Process scheduling |
| Per-process heap (bumpalo) | heap.rs |
Term allocation |
| Copying GC | heap.rs |
Memory reclamation |
| Tagged terms (64-bit) | term.rs |
Value representation |
| Atom table (global, lock-free reads) | atom_table.rs |
Atom interning |
| Message passing (deep copy) | message.rs |
Inter-process communication |
| Selective receive | process.rs |
Mailbox scanning |
| Process spawning | ffi.rs |
spawn/3 |
| Socket registry | socket.rs |
gen_tcp handle management |
These primitives are all fully operational, enabling OTP behaviours (Tier 3):
| Primitive | Why Rust? |
|---|---|
register/2, whereis/1, unregister/1 |
Global process name table, needs atomic access |
link/1, unlink/1 |
Bidirectional link sets on each process, checked at exit |
process_flag(trap_exit, true) |
Per-process flag, changes exit signal behavior |
| EXIT signal propagation | Scheduler must deliver signals when a process dies |
monitor/2, demonitor/1 |
Unidirectional monitoring with DOWN message delivery |
process_info/1,2 |
Introspection into process internals |
put/2, get/1 (process dictionary) |
Per-process mutable state, needs GC root tracking |
erlang:send_after/3, timers |
Timer wheel integrated with scheduler wake-up |
| ETS tables | Shared mutable storage with concurrent access |
The dependency chain for OTP support:
register/whereis ─┐
link/trap_exit ├──► gen_server.erl (Tier 3)
monitor/demonitor ─┤
EXIT signals ─┘
│
▼
supervisor.erl (Tier 3)
These are implemented in Rust but are not coupled to the runtime core. They are pure (or nearly pure) functions: they take Erlang terms as input, call a Rust crate, and return Erlang terms. They don't touch the scheduler, don't manage processes, and don't need GC integration beyond allocating results on the process heap.
Two reasons:
-
Performance:
serde_jsonparses JSON 10-100x faster than any parser written in compiled Erlang.ringcomputes SHA-256 hundreds of times faster. These are domains where native code with SIMD, cache-friendly layouts, and zero-allocation parsing matter. -
Ecosystem: the Rust crate ecosystem has production-grade implementations of JSON, regex, crypto, HTTP, compression, and more. Reimplementing them in Erlang would be a multi-year effort with inferior results.
This matches what BEAM does: crypto is backed by OpenSSL via NIFs, re
is backed by PCRE via NIFs, zlib is backed by zlib via a port driver.
nera uses Rust crates instead of C libraries.
| Module | Rust Crate | API Surface | Priority |
|---|---|---|---|
json |
serde_json |
encode/1, decode/1 |
High |
base64 |
base64 |
encode/1, decode/1 |
High |
re |
regex |
run/2,3, replace/3,4, split/2,3 |
High |
file |
std::fs |
read_file/1, write_file/2, list_dir/1 |
High |
crypto |
ring or sha2+hmac |
hash/2, mac/3,4 |
Medium |
unicode |
unicode-segmentation |
characters_to_binary/1,2 |
Medium |
filename |
std::path |
join/1,2, basename/1, dirname/1 |
Medium |
os |
std::env |
getenv/1, cmd/1 |
Medium |
zlib |
flate2 |
compress/1, uncompress/1 |
Low |
http_client |
ureq (sync) |
get/1, post/2,3 |
Low |
rand |
rand |
uniform/0,1, seed/1 |
Low |
calendar |
chrono or time |
local_time/0, universal_time/0 |
Low |
Every Tier 2 module follows the same four-step pattern, already established
by gen_tcp and socket.rs:
Pure Rust logic, no FFI boilerplate. Works with ProcessHeap and ErlTerm
directly:
// crates/nera-runtime/src/json.rs
use serde_json::Value;
/// Encode an ErlTerm (map/list/number/binary/atom) to a JSON binary.
pub fn encode(heap: &mut ProcessHeap, term: ErlTerm) -> Result<ErlTerm, ErlTerm> {
let value = term_to_json(term)?;
let bytes = serde_json::to_vec(&value).map_err(|e| make_error(heap, e))?;
Ok(heap.alloc_binary(&bytes))
}
/// Decode a JSON binary into an ErlTerm.
pub fn decode(heap: &mut ProcessHeap, bin: ErlTerm) -> Result<ErlTerm, ErlTerm> {
let bytes = bin.as_binary_bytes()?;
let value: Value = serde_json::from_slice(bytes).map_err(|e| make_error(heap, e))?;
Ok(json_to_term(heap, &value))
}Term-to-JSON mapping:
| Erlang type | JSON type |
|---|---|
#{<<"key">> => Val} (map with binary keys) |
object |
[1, 2, 3] (list) |
array |
| Small integer / float | number |
<<"text">> (binary) |
string |
true / false |
boolean |
null |
null |
Thin C-ABI function, same convention as all existing FFI:
#[no_mangle]
pub extern "C" fn nera_rt_json_encode(ctx: *mut ProcessContext, term: u64) -> u64 {
let heap = unsafe { &mut *(*ctx).heap_ptr };
match json::encode(heap, ErlTerm(term)) {
Ok(result) => make_ok_tuple(heap, result).0,
Err(reason) => make_error_tuple(heap, reason).0,
}
}
#[no_mangle]
pub extern "C" fn nera_rt_json_decode(ctx: *mut ProcessContext, bin: u64) -> u64 {
let heap = unsafe { &mut *(*ctx).heap_ptr };
match json::decode(heap, ErlTerm(bin)) {
Ok(result) => make_ok_tuple(heap, result).0,
Err(reason) => make_error_tuple(heap, reason).0,
}
}All BIF metadata lives in a unified table (nera-ir/src/bif_table.rs). One entry per function — IR lowering and codegen both consume it automatically:
("json", "encode", 1) => Some(RuntimeFunc {
symbol: "nera_rt_json_encode",
arity: 1,
returns_via_ctx: false,
}),
("json", "decode", 1) => Some(RuntimeFunc {
symbol: "nera_rt_json_decode",
arity: 1,
returns_via_ctx: false,
}),-module(my_api).
-export([handle_request/1]).
handle_request(Body) ->
case json:decode(Body) of
{ok, #{<<"action">> := Action}} ->
Response = #{status => ok, action => Action},
{ok, json:encode(Response)};
{error, Reason} ->
{error, Reason}
end.No special syntax. No imports. The module name (json) resolves to a Rust
implementation at compile time, exactly like io:format or gen_tcp:listen
do today.
This is not a general-purpose FFI/NIF system for end users. It is the compiler's own standard library, implemented in Rust for pragmatic reasons. There is no plugin mechanism, no dynamic loading, no user-extensible dispatch. The set of Tier 2 modules is fixed at compile time.
A user-facing FFI system (analogous to BEAM NIFs) is a separate concern,
documented in NERA_NIF_DESIGN.md.
These are .erl files compiled by nera alongside user code. They express
application-level patterns that are naturally built on message passing,
pattern matching, and process management — the things Erlang excels at.
OTP behaviours are not magic. gen_server is approximately 200 lines of
Erlang: a receive loop that dispatches call, cast, and info
messages to callback functions. Hardcoding this in Rust would mean:
- Combinatorial explosion: gen_server, gen_statem, gen_event, supervisor, application, proc_lib, sys — each with many edge cases.
- Semantics drift: subtle incompatibilities with real OTP that confuse experienced Erlang developers.
- Rigidity: users cannot customize, extend, or inspect the implementation. In real Erlang, you can read gen_server source, copy it, modify it.
This matches BEAM's own design: the VM provides link/1, monitor/2,
register/2 as BIFs (our Tier 1), and all of OTP is built on top in
Erlang (our Tier 3).
These are implemented and shipped in erlang/stdlib/:
| Module | Depends On (Tier 1) | Description |
|---|---|---|
proc_lib |
spawn, links, EXIT signals | "Safe" process spawning with crash reports ✅ |
gen_server |
register, monitor, links, trap_exit | Generic server behaviour ✅ |
gen_statem |
same as gen_server | Generic state machine |
supervisor |
gen_server, links, monitors | Process supervision trees ✅ |
application |
supervisor | Application start/stop lifecycle ✅ |
sys |
gen_server message protocol | Debug and introspection for OTP processes |
gen_event |
gen_server | Generic event handler |
lists |
none (pure) | List utilities (map, foldl, filter, ...) ✅ |
proplists |
none (pure) | Property list utilities |
string |
none (pure) | String manipulation |
dict / orddict |
none (pure) | Dictionary data structures |
queue |
none (pure) | Double-ended queue |
timer |
send_after, spawn | Timer utilities (apply_after, send_interval) ✅ |
io_lib |
none (pure) | I/O formatting utilities |
Note on pure modules: lists, string, proplists, and similar
modules contain no process interaction. They could theoretically be Tier 2
(Rust), but they are naturally and efficiently expressed in Erlang. A
compiled lists:map/2 over a 1000-element list runs in microseconds — fast
enough that Rust adds no practical benefit. These modules also serve as a
test of nera's own compilation: if lists.erl compiles and runs
correctly, the compiler is sound for real-world code.
Implemented: Option A (stdlib directory) is the current approach. Stdlib .erl files live in erlang/stdlib/ and are auto-discovered via find_stdlib_dir() — which checks installed layout (<exe>/../lib/nera/erlang/stdlib/), workspace layout (<exe>/../../erlang/stdlib/), and CWD fallback. User modules shadow stdlib by name. The --no-stdlib flag disables auto-inclusion.
Tier 3 modules are compiled the same way as user modules. There are two possible integration strategies:
Option A: Stdlib directory
nera ships with a lib/ directory of .erl files. The compiler
automatically includes them when their module name is referenced:
nera/
lib/
gen_server.erl
supervisor.erl
lists.erl
...
$ nera my_app.erl -o my_app
# Compiler sees gen_server:start_link → includes lib/gen_server.erlOption B: Pre-compiled object files
The stdlib modules are pre-compiled to .o files during cargo build. The
linker includes them when referenced:
nera/
lib/
gen_server.o # Pre-compiled by nera during build
supervisor.o
lists.o
Option A is simpler and allows easier development/debugging. Option B is faster for large projects since stdlib modules are compiled once. Both can coexist — start with A, add B as an optimization.
The three-tier split mirrors how BEAM/OTP is structured, with Rust replacing C:
| Layer | BEAM/OTP | nera |
|---|---|---|
| Runtime primitives | C code in erts (link, monitor, ETS, scheduler) | Rust in nera-runtime (Tier 1) |
| Native library bindings | C NIFs/port drivers (crypto→OpenSSL, re→PCRE, zlib→zlib) | Rust extensions (crypto→ring, re→regex, json→serde_json) (Tier 2) |
| OTP frameworks | Erlang modules (gen_server.erl, supervisor.erl) | Erlang modules compiled by nera (Tier 3) |
The advantage of nera's approach is lower risk, not containment. Tier 2 modules use Rust crates, mostly written in safe Rust, instead of C libraries accessed through the NIF API. A bug in safe Rust code cannot cause the buffer overflows, use-after-free or data races that make a C parser or TLS library dangerous. That Rust code still runs in the same process and address space as the runtime and every Erlang process, so it is not isolated from them, and a native failure can still stop the whole program. The next section states these limits.
These limits apply to all native code in a nera program: Tier 1 primitives, Tier 2 modules and the crates they use.
Rust reduces risk. In code written in safe Rust, the language prevents
memory-safety bugs such as out-of-bounds writes, use-after-free and data
races: an out-of-range index panics instead of corrupting memory. Invalid
input can be reported as an error (badarg, {error, Reason}), provided the
code checks it before it can panic.
Rust does not contain native code. Tier 1 and Tier 2 code is linked into the same executable and runs on the same scheduler threads as compiled Erlang code. No sandbox, separate OS process or memory protection separates a Tier 2 module from the runtime or from other Erlang processes. Safe Rust's guarantee ends at:
unsafeblocks. The runtime uses them for its term representation and heaps, and in nearly every C-ABI entry point called by generated code (thenera_rt_json_encodeexample above dereferences a raw context pointer).- Dependencies that contain
unsafecode or code in other languages, such as the C jemalloc allocator linked on non-MSVC targets. - The calling convention between generated code and the runtime.
A defect in any of these can corrupt memory that belongs to the runtime or to any Erlang process. Rust also does not limit how much memory or processor time a native call uses.
Nothing guarantees the runtime keeps running. An Erlang exception ends one
process, and links and monitors report it. A native failure is not reported
that way: no EXIT or DOWN message is delivered. Usually it ends the whole
OS process, and every Erlang process stops with it. Today the runtime does not
catch Rust panics. Its BIF entry points are extern "C" functions, so a panic
that reaches one aborts the program. A panic in runtime code that no
extern "C" frame encloses, such as garbage collection in a scheduler's own
loop or the timer and I/O threads, instead ends only that thread and can leave
the Erlang processes that depend on it stalled. Catching panics would not
change the following, which also stop the program:
- memory allocation failure (Rust aborts by default);
- stack overflow on a native thread;
- a panic raised while another panic is unwinding;
- an explicit abort;
- undefined behaviour in
unsafeor foreign code, which may instead corrupt memory silently and fail later.
Tier 2 code therefore still has to validate its input, return errors instead
of panicking, and bound its allocations and work. Keep its unsafe code small
and reviewed. Rust makes a runtime crash less likely; it neither makes one
impossible nor confines one to the calling Erlang process.
Because Tier 2 modules are part of the compiler's own build, nera can offer batteries-included functionality that OTP historically left to third-party packages:
| Capability | OTP stdlib | nera stdlib |
|---|---|---|
| JSON | Added in OTP 27 (pure Erlang) | serde_json (10-100x faster) |
| HTTP client | httpc (dated, limited) |
ureq (modern, TLS built-in) |
| Regex | re (PCRE via NIF) |
regex (linear time, no backtracking DoS) |
| Crypto | crypto (OpenSSL via NIF) |
ring (Rust, C and assembly) |
| Compression | zlib (C via port driver) |
flate2 (Rust; backend may use unsafe or C) |
| UUID | not in stdlib | uuid crate |
| Base64 | base64 (pure Erlang) |
base64 crate (SIMD-accelerated) |
crates/nera-runtime/src/
lib.rs
term.rs # Tier 1: term representation
heap.rs # Tier 1: per-process heap + GC
scheduler.rs # Tier 1: work-stealing scheduler
process.rs # Tier 1: process state, mailbox
message.rs # Tier 1: deep-copy message passing
atom_table.rs # Tier 1: global atom table
ffi.rs # FFI wrappers for both Tier 1 and Tier 2
bif.rs # BIF implementations
socket.rs # Tier 2: gen_tcp backend (existing)
entry.rs # Tier 1: C-ABI entry point
# --- Tier 2 extension modules ---
json.rs # json:encode/1, json:decode/1
re_ext.rs # re:run/2, re:replace/3
crypto_ext.rs # crypto:hash/2, crypto:mac/3
base64_ext.rs # base64:encode/1, base64:decode/1
file_ext.rs # file:read_file/1, file:write_file/2
os_ext.rs # os:getenv/1, os:cmd/1
crates/nera-codegen/src/
builtins.rs # Maps module:function → FFI symbol (Tier 1 + Tier 2)
lib/ # Tier 3: Erlang stdlib modules
lists.erl
proplists.erl
gen_server.erl
supervisor.erl
proc_lib.erl
string.erl
io_lib.erl
queue.erl
Add high-value Rust extension modules that work today:
json—serde_json. High demand, immediate value.base64— trivial to add, constant use in web/networking.file—read_file/1,write_file/2. Basic file I/O.re—regexcrate. Needed for string processing.
Add the runtime primitives that OTP behaviours require:
register/2,whereis/1— process registrylink/1,process_flag(trap_exit, true)— bidirectional links- EXIT signal propagation — scheduler-level
monitor/2,demonitor/1— unidirectional monitorsput/2,get/1— process dictionaryerlang:send_after/3— timer support
Build OTP behaviours as Erlang modules:
proc_lib.erl— safe spawning with crash reportsgen_server.erl— generic serversupervisor.erl— process supervisionlists.erl— standard list operations
Add remaining extension modules as demand arises:
crypto,unicode,zlib,http_client,rand,calendar
The guiding principle is: implement each piece at the lowest tier that makes sense.
- Default for new stdlib APIs → Tier 3 (Erlang modules)
- If it touches scheduler/runtime internals → Tier 1 (Rust, runtime core)
- If Rust provides a measured material win or unavailable capability → Tier 2 (Rust, extension library)
This gives nera a "batteries included" standard library that is less exposed to memory-safety bugs (Rust instead of C NIFs, although native code is not isolated and can still stop the runtime; see What Rust Does and Does Not Guarantee), faster (serde_json instead of pure Erlang JSON), and idiomatically correct (gen_server is real Erlang, not a Rust reimagination of what gen_server might look like).