Skip to content

fix(tn-reth): derive per-worker RPC/IPC endpoints in start_rpc (#1287) - #1298

Open
MavenRain wants to merge 8 commits into
mainfrom
tn-1287-per-worker-rpc-endpoints
Open

fix(tn-reth): derive per-worker RPC/IPC endpoints in start_rpc (#1287)#1298
MavenRain wants to merge 8 commits into
mainfrom
tn-1287-per-worker-rpc-endpoints

Conversation

@MavenRain

@MavenRain MavenRain commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Closes #1287. (Cantina #23)

Problem

RethEnv::start_rpc read the one process-wide NodeConfig.rpc on every call, and ExecutionNodeInner::initialize_worker_components calls it once per worker. With more than one worker (the #554 to #559 track) every worker's RPC server targeted the same http/ws socket and the same IPC path:

  • IPC first: reth unlinks the endpoint path before it binds, so the second worker silently stole worker 0's socket file. The default configuration (IPC only) started cleanly and served only the last worker's pool, with nothing above debug!.
  • http/ws second: a fixed non-zero port made the second bind fail with AddrInUse, which initialize_worker_components propagates as a fatal startup error, after the IPC steal already happened.

--with-unused-ports did not close the gap: it randomizes the IPC path once per process, so every worker still shared it. No attacker is required for either failure; the default configuration failed silently rather than loudly.

Not a bug on main today (only DEFAULT_WORKER_ID is ever initialized). This is the design gap #558 has to close, landed as its precursor.

Fix

  • crates/tn-reth/src/env/rpc.rs: start_rpc takes the worker_id and derives that worker's RpcServerArgs from the operator's config (worker_rpc_server_args) instead of reusing the shared one:
    • worker 0 keeps the operator's values unchanged, so single-worker nodes and existing tooling see exactly the configured endpoints;
    • worker w > 0 shifts an enabled http port down by 200 * w, shifts an enabled ws port up by 400 * w, and suffixes an enabled IPC path with -w{w};
    • the stride is derived from reth's real --instance bound, not guessed: instance offsets span at most 199 ports (instance <= 200 in reth v1.11.3), so a 200-port worker stride keeps every (instance, worker) pair on a distinct port, and the -w{w} IPC suffix is disjoint from instance's -{i} by construction;
    • port 0 (the --with-unused-ports sentinel) passes through, since the OS assigns a distinct port per bind, and the flag's one random IPC path still gets the per-worker suffix;
    • a shift that leaves the valid range fails startup loudly (TnRethError::WorkerRpcPort) instead of wrapping into a port another worker or instance owns; a derived http port additionally stops above the privileged range (below 1024 a non-root process cannot bind), so that failure also names the worker, base, and offset instead of dying later in the bind with a bare permission error;
    • when both transports are enabled on fixed ports the derivation requires ws_port >= http_port (reth's default layout; equality is the shared http+ws server): http bands stride down and ws bands stride up, so inverted bases would let one worker's http band land on another worker's ws band; the first derived worker fails loudly (TnRethError::WorkerRpcPortOrder) and worker 0 still binds inverted bases as configured, keeping single-worker nodes working;
    • a disabled transport keeps its configured value and can never error.
  • crates/tn-reth/src/env/rpc.rs: the resolved endpoints (http/ws/ipc) are logged per worker at info! once the server starts.
  • crates/node/src/engine/inner.rs: initialize_worker_components passes its worker_id to start_rpc.
  • crates/tn-reth/src/error.rs: new WorkerRpcPort and WorkerRpcPortOrder variants carrying the worker id (WorkerId), the transport, and the offending ports and offset.

No wire change: RpcNodeInfo carries no endpoints.

Testing

  • test_worker_zero_keeps_operator_rpc_endpoints: worker 0's derived args equal the operator's.
  • test_worker_endpoints_are_distinct_per_worker: workers 1 and 2 derive distinct http/ws ports and IPC paths with the exact band arithmetic.
  • test_worker_bands_clear_the_instance_range: the closest approach between the worker and instance schemes (worker 1 of instance 1 against worker 0 of instance 200) stays separated on both transports.
  • test_out_of_range_worker_port_is_a_loud_error: an enabled http port below the band and an enabled ws port at u16::MAX both fail with WorkerRpcPort, not a wrapped port.
  • test_zero_ports_stay_os_assigned_and_ipc_still_suffixes: --with-unused-ports semantics survive derivation.
  • test_inverted_transport_bases_are_a_loud_error: fixed ws below fixed http fails the first derived worker with WorkerRpcPortOrder; worker 0 keeps the inverted config as given.
  • test_equal_transport_bases_stay_valid: reth's shared http+ws server (equal ports) derives non-colliding split bands.
  • test_http_band_below_the_privileged_floor_is_a_loud_error: a derived http port under 1024 fails at derivation time, not at bind time.
  • test_disabled_transports_keep_their_configured_values: a disabled transport never shifts and never errors; disabled IPC keeps its path.
  • test_two_workers_bind_distinct_live_ipc_sockets (unix): two workers started on one RethEnv leave both IPC socket files on disk; before this change the second start unlinked worker 0's.

Ran nightly cargo fmt -- --check, cargo check --workspace --all-targets, and cargo nextest run -p tn-reth on the rpc tests locally. Each new load-bearing test was seen to fail with the derivation reverted to the shared config and pass with it restored.

Every worker started the one shared NodeConfig.rpc: the second worker
silently unlinked worker 0's IPC socket (reth removes the endpoint path
before it binds) and a fixed http/ws port failed startup with AddrInUse.
Derive each worker's RpcServerArgs instead: worker 0 keeps the operator's
values; worker w shifts an enabled http port down by 200*w and an enabled
ws port up by 400*w (the stride clears reth's whole --instance range) and
suffixes an enabled IPC path with -w{w}. Port 0 stays OS-assigned, a
shift that leaves the valid range fails startup loudly, and the resolved
endpoints are logged per worker at info.

Cross-transport distinctness is enforced up front: fixed bases need
ws_port >= http_port (equality is reth's shared http+ws server), since
inverted bases would let one worker's http band land on another worker's
ws band, and a derived http port stops above the privileged range so the
failure names the worker and offset instead of a bare EACCES at bind.

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
@MavenRain MavenRain self-assigned this Aug 27, 2026
@MavenRain
MavenRain deployed to merge-into-main August 27, 2026 17:46 — with GitHub Actions Active
…c-endpoints

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…c-endpoints

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…c-endpoints

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…c-endpoints

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
…c-endpoints

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
grantkee
grantkee previously approved these changes Sep 4, 2026

@grantkee grantkee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

…c-endpoints

Round 6: main 66e0d14, 8 commits past the round-5 target 87364b2 (#1270,
#1295, #1291, #1288, #1280, #1268, #1294 and one fork merge; 38 files).

Two files overlap the PR. engine/inner.rs auto-merged: main's WorkerBaseFee
handle at get_rpc_server, the PR's worker_id at start_rpc. env/rpc.rs needed
two hand reconciliations because #1295 changed get_rpc_server to take a
WorkerBaseFee handle instead of a BaseFeeContainer clone:

- The tn_types import: keep the PR's WorkerId and take main's WorkerBaseFee.
- The PR's start_worker_rpc test helper: take a &GasAccumulator and pass
  worker_id's container to init_txn_pool and its worker_base_fee handle to
  get_rpc_server. The two-worker IPC test builds one GasAccumulator::new(2)
  so each worker resolves its own slot, matching main's
  fee_history_methods_for_worker helper.

Every other file is byte-identical to origin/main.

Signed-off-by: Onyeka Obi <softwareengineerasaservant@isurvivable.cv>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants