No supported way for a bundle to learn which node it is running on after CELLD_VAR_* removal

[samquarm]

We run a self-hosted multi-region fleet on v0.4.1 and are blocked from moving to v0.5.0 by the removal of CELLD_VAR_*. I want to describe the use case in case it isn't one you had in view, and be concrete about the cost. I'm not arguing the removal was wrong.

The requirement: one string per node

A bundle needs to know the name of the node it is running on. Nothing else.

Clients connect to a per-region endpoint that terminates on the node for that region. A short-lived credential is minted by our control plane and bound to the region it was minted for, so the serving node can reject one minted elsewhere:

// simplified
const ok = verifyCredential(secret, token, {
  subject,
  region: env.REGION,   // the only per-node value we need
});
if (!ok) return deny();

This is defence in depth on top of the signature and subject checks. It stops a validly minted credential being replayed against a different region.

The shared secret is fleet-wide and moves to vars cleanly. The region name is the problem, because it is the one value that differs per node.

Why the documented replacement doesn't cover it

vars is deployment-wide, and a fleet runs one application with one deploy/current.json, so there is no per-node override. We also can't infer the node from the request: docs/cloudflare-compat.md states the inbound Request has "a cf object without Cloudflare edge fields".

On v0.4.1 there was a second route, globalThis.__cell.node. v0.5.0 closes it deliberately (docs/security.md: "a bundle cannot name a host function, and globalThis carries no __-prefixed property"). We think that hardening is correct and are not asking for it back. We confirmed it is genuinely closed: on v0.5.0, typeof globalThis.__celld is "undefined" from bundle code, both under celld dev and on a real fleet node.

We understand the model behind it. bootstrap.rs says "Every cell scope routes through the host, whichever node owns it", so a bundle asking where it runs in order to make placement decisions is a category error.

Our use is narrower than placement. We do not want node identity to decide where anything goes, since request routing happens entirely in our own control plane before celld is involved. We want it only so a node can refuse a credential issued for somewhere else.

The broader shape: Durable Objects at the edge

Stepping back from our own case, this seems inherent to what celld is for.

The product is self-hosted, distributed Durable Objects. The natural way to deploy that is the way Cloudflare deploys the thing it replicates: many nodes spread across locations, each one a distinct place. Operators adopting celld to get DO semantics without DO billing will tend to scale by adding nodes in more locations, not by making one node bigger. Every node added makes per-node identity more useful, not less.

There is also a capability gap worth naming. On Cloudflare, a Worker can learn something about where it is executing from request.cf. On celld the cf object carries no edge fields, which is entirely reasonable since celld can't prove geolocation it doesn't have. But the combination is what bites: celld provides no edge context on the request, and v0.5.0 removed the last remaining way for a bundle to learn its own node. A self-hosted fleet therefore has strictly less location awareness available to application code than the platform it is replicating, with no supported path to add it back.

We don't think the answer is to synthesise cf fields, which would be claiming knowledge celld doesn't have. A node's own name, which celld does know for certain and already injects into the isolate, seems like the honest primitive: not geolocation, not placement input, just "which of the operator's nodes am I".

The cost of staying on v0.4.1

docs/security.md states that "Security fixes apply to the latest release only, so older alpha builds do not receive fixes". So pinning is not merely forgoing features. It means knowingly forgoing security fixes on a fleet that terminates public traffic. That is what makes this urgent rather than cosmetic for us.

What would unblock us

Any supported way for a bundle to learn its own node name. Roughly in order of how well each seems to fit your model:

1. Expose the node id on env. You already compute and inject it (bootstrap.rs, inject_routing), so it is close to hand. Read-only, no new configuration surface, and it cannot be used to influence placement.
2. A ctx field such as ctx.node, if env is meant to stay purely config-derived.
3. Restore the per-node overlay, but opt-in. Details below.

On option 3, with a working implementation

We prototyped this against v0.5.0 to check it was feasible before suggesting it. Roughly 60 lines across env_vars.rs and fleet.rs, gated behind an explicit CELLD_ALLOW_NODE_VARS, default off.

Opt-in specifically answers the reason you gave for the removal. env_vars.rs explains that a node started with a silently ignored CELLD_VAR_ "would run with no override at all, which reads inside a Worker as a missing secret with nothing in the log", which is a real and nasty failure mode. An operator who had to explicitly enable the feature cannot be surprised by it, so the default stays safe and the sharp edge is only reachable by someone who asked for it.

Verified behaviour of the prototype:

case	result
CELLD_VAR_ set, no opt-in	hard fail, current message unchanged
CELLD_VAR_ set, opt-in =1	proceeds, value arrives in env
CELLD_VAR_ set, opt-in =0	hard fail
opt-in set to a malformed value	loud reject via the existing parse_flag, must be 0 or 1
no CELLD_VAR_, opt-in on	unchanged

Two details we found worth getting right, in case they're useful regardless of what you decide:

• The overlay must read std::env::vars_os(), not vars(). The latter panics if any name or value in the environment is not valid unicode, including variables unrelated to celld. validate() already uses vars_os().
• The opt-in itself should go through flag() rather than a lenient match, so a typo is rejected loudly rather than silently leaving the feature off.

We deliberately did not restore CELLD_VARS_FILE. One string per node is the need, and leaving the file loader out keeps worker_vars()'s signature and its single caller unchanged.

Happy to share the diff if it is useful, or to test any of the three options on a real multi-region fleet and report back with per-node timing and lease behaviour.

Environment

• celld v0.4.1 in production, v0.5.0 for the prototype above
• Multi-region self-hosted fleet, Linux x86_64 and aarch64
• S3-compatible object store with conditional write support
• Cell classes holding hibernatable WebSockets (acceptWebSocket, getWebSockets, serializeAttachment); one also uses ctx.storage

[ry]

There's no equivalent CF api for this so I'm hesitant to add it.

[samquarm]

How we use celld, and what the per-node var is for

We run celld as a self-hosted Durable Objects replacement across a ten-node,
multi-region fleet spanning several continents. It backs per-user realtime
inboxes and per-community fanout channels. The same worker bundle is deployed
to every node.

Why we need a per-node variable

Each node has to know which region it is. Our bundle reads CELLD_REGION at
startup and uses it to mint and validate realtime tickets. Without it the
bundle cannot identify itself and returns 503.

On 0.4.x we set that with the CELLD_VAR_<NAME> overlay: one
CELLD_VAR_CELLD_REGION=<region> per node, and the bundle sees CELLD_REGION
set accordingly. One string per node, with the binary, the bundle and every
other setting identical fleet-wide.

0.5.0 removed the overlay and fails at startup when CELLD_VAR_* is present.

The patch, already forked and already running

We have forked 0.5.0, added this, and it is live on all ten nodes. It is not a
proposal we are asking you to evaluate on paper.

About 60 lines across two files. It restores the overlay behind an explicit
CELLD_ALLOW_NODE_VARS=1, default OFF. Without the opt-in, 0.5.0 behaves
exactly as it does today, including the startup failure, so nobody who has not
asked for this sees any change.

We deliberately did not restore CELLD_VARS_FILE. One string per node is all
we need, and omitting it leaves worker_vars()'s signature and its single
caller untouched, which keeps the merge small.

Running on 10/10 nodes on patched 0.5.0: zero restarts, health 200 on every
node, and the per-node identity confirmed reaching the bundle.

We are happy to keep carrying the fork. We would rather not, mainly because a
default-OFF opt-in seemed like the kind of thing that costs you nothing and
saves every self-hoster the same 60 lines.

Happy to share the code. We can open a PR against 0.5.0 with the patch as
it stands, or just paste the diff here if you would rather look before there is
a PR open. Either way it is yours to take, modify or reject, and we are not
precious about the shape: if you want it written differently, or gated
differently, or named something else, tell us what you would accept and we will
send that instead.

We can also share how we exercised it, since an opt-in like this is mostly a
question of proving it does not change the default path. Our gate matrix covers
the opt-in on and off, the variable present and absent, and the 0.5.0 startup
failure still firing untouched when the flag is not set.

Why a celld node needs this when a Cloudflare Worker does not

This may be the actual shape of the gap, and it could explain why there is no
equivalent API to point at.

On Cloudflare, placement is Cloudflare's job. An isolate never needs to be told
where it is, because it did not choose and cannot influence it, and anything it
does need arrives per request as request.cf.colo.

Self-hosting moves placement to the operator. We decide which machine serves
which region, so the process has to be told something Cloudflare would never
have to tell it.

So it is not that Cloudflare has no notion of "where am I". It does, through
request.cf.*. It is that Cloudflare delivers it per request, because the
platform owns placement, while a self-hosted fleet needs it per process,
because the operator does.

How we route to the nearest node, for context

A Cloudflare Worker at the edge picks the node per request: cf.colo first,
cf.country as a fallback, and a global default when neither resolves.

colo deliberately outranks country. The hop being optimised is worker to
node, and the worker executes at the colo, whereas country is where the user
is. Those two diverge exactly where the cost is highest. Traffic from some
countries is routinely backhauled to PoPs on another continent, so country
routing would dispatch a worker thousands of kilometres to a node in the user's
home region while a far closer node sat in the same building as the executing
isolate.

R2 Local Uploads, worth knowing about, plus one open question for you

Unrelated to the patch, but probably the most useful thing we can hand back.

celld uses the bucket as its consensus substrate, and a bucket has a single
immutable location hint. Our writes originate from ten nodes across several
continents, so every node lease renewal paid the leg from its local PoP to the
bucket's home region. That is the textbook worst case for this design, and it
dominated our fencing behaviour.

Enabling R2 Local Uploads (Beta) on the celld bucket
(wrangler r2 bucket local-uploads enable <bucket>) removed most of it. On our
worst node, the conditional write that the node lease performs went from
roughly 1.7s to 875 ms, and plain PUT p50 to 905 ms over 25 samples. A
hardware change on that node landed in the same window and contributed roughly
as much, so attribute about half of that to Local Uploads rather than all of it.

The open question, and the reason we are raising it here rather than just
being pleased. The dashboard only advertises "improve performance for
cross-region uploads", but wrangler r2 bucket local-uploads get spells out
the semantics:

"Object data is written to the nearest region first, then asynchronously
replicated to the bucket's primary region."

That is eventual consistency on the data path, underneath a consensus
substrate. The concrete worry: node A writes its lease locally, node B reads
that lease from another continent to decide whether A is alive. If replication
lag ever exceeded the read, a peer could see a stale lease and conclude a
healthy node is dead.

What we verified across five nodes: a deliberately stale etag still returns 412
everywhere, so the conditional-write contract is still serialised against
something authoritative. What we did not verify: cross-region read freshness.
Those are different guarantees, and celld appears to need both.

So, the question for you. Is a Local-Uploads-enabled bucket a configuration
celld can safely sit on, or does the async replication break an assumption in
the lease and fencing design that we would only discover under partition? We
are watching our fence count and would disable it on the celld bucket first if
it rises, but you would know from the design side far faster than we will learn
it from the metrics.

If the env-var shape is the objection

Would you be more open to exposing node identity at runtime instead, something
on the request or context mirroring request.cf.colo, rather than as an
environment overlay? That is the same information in the shape Cloudflare
already uses, it stays closer to the Durable Objects model you are tracking,
and it would solve this for us just as well.

We are glad to do that work rather than hand you a design and wait. Point us at
the shape you want and we will write it, test it on the fleet we already have
running, and send you the results alongside the patch.

[dacuna-ic]

We're also having issues with the new approach. Currently our services decouple deployment from env changes, meaning we'd now need a celld deploy in order to propagate any env var changes, instead of relying on setting the env var in the celld service.

I think an opt-in to CELLD_VAR_* variables would be great, but I'm not too sure what was the rationale for the removal, it was a genuinely useful way to set these values.