WIP
PurrBalancer registers the rooms and lets user know of all servers
PurrLay is the relay and handles the actual game and connections
PurrBalancer and every PurrLay relay use the same routing SECRET. Local and
self-hosted trusted-coop setups are zero-config and default to PURRNET; set
SECRET on both services to override it. Values must be printable ASCII without
spaces because they travel in an HTTP header. The production Fly workflow still
requires an explicit shared value so unrelated hosted deployments do not all use
the well-known default.
PurrBalancer and PurrLay are one protocol pair. The production workflow deploys relays one at a time before deploying the balancer. A new relay accepts the old balancer's registration response without repeatedly republishing rooms; once the new balancer starts, it explicitly asks every current relay boot to republish its authoritative room snapshot.
The workflow runs the complete test suite, keeps one shared service credential, and verifies each relay's live boot ID before advancing. After the balancer is deployed, it waits for the exact live relay boots to register and smoke-tests room allocation and join. Workflow concurrency serializes production rollouts.
The apps currently run one Machine each, so replacing an individual relay can briefly interrupt sessions on that relay. Eliminating that per-relay restart gap would require overlapping Machines and machine-specific health/routing; it does not require a fleet-wide scale-to-zero maintenance barrier.
Every relay process creates an opaque boot ID and includes it in RegisterServer and every room publication alongside its endpoint. The relay listener starts before registration, and its /ping response proves the live boot ID. The balancer serializes endpoint proofs and accepts a different boot only when that endpoint currently reports the incoming ID; therefore a delayed heartbeat from the prior process cannot reclaim a restarted endpoint. Routing metadata is immutable within one boot: a heartbeat that changes host, region, endpoint, or ports is rejected and must use a new boot ID. A confirmed replacement atomically invalidates that endpoint's old room routes and requires the new process to republish, while delayed room publications from the prior process are rejected. Health checks verify the exact boot ID and a balancer-owned liveness version, so a failed probe captured before a newer heartbeat cannot remove the refreshed relay.
Within one relay process, every room publication carries an opaque room epoch and a room-owned monotonic revision. Room registration is an atomic name-ownership compare-and-set: a different epoch cannot replace a live room, while a released or expired name may be acquired by a new epoch. Count updates and unregister requests are accepted only for the owning relay boot, current epoch, and a newer revision, so delayed HTTP requests cannot remove or overwrite a newer route.
The relay republishes authoritative room counts with its 30-second balancer registration heartbeat and reacquires all room registrations when the balancer reports a new relay registration or boot. This repairs a dropped last update, balancer restart, or transient health-check removal without resetting an empty room's original expiry time. A locally removed room first enters an unregister outbox keyed by its exact name, epoch, and revision. Failed sends remain there and are retried on every registration heartbeat; an entry is removed only after the balancer confirms it was applied or safely ignored because that exact removal was already applied or a newer room incarnation owns the name.
The balancer retains released publication tombstones for a bounded safety horizon. ROOM_PUBLICATION_TOMBSTONE_RETENTION_SECONDS defaults to 3600 seconds; non-positive values fall back to that default. Cleanup runs with the 30-second empty-room cleanup tick and rechecks expiry under the same per-room route lease before deleting. During the horizon, a delayed registration for the removed epoch remains fenced. After deletion, that bounded guarantee ends and a sufficiently delayed old registration can be accepted if no newer live publication exists. Configure the horizon longer than the maximum relay publication/outbox retry delay, including deployment and outage margin, and keep the balancer's UTC clock stable because expiry is wall-clock based.
Room allocation is a mutating POST /allocate_ws. PurrNet creates one opaque allocation_id outside its retry loop and the balancer forwards it unchanged. An exact replay returns the original allocation capability; a different ID cannot take over that allocation. For allocation and join, the balancer validates the complete relay connection capability and injects the selected relay's authoritative host; PurrNet connects to that returned host instead of relying on its earlier regional probe. Malformed or incomplete relay capabilities return a sanitized 502 Bad Gateway. The relay likewise retries balancer registration with the exact same epoch/revision when a response is lost. If the outcome remains unknown, it retains that pending identity and a later exact allocation resumes the idempotent registration instead of creating a competing epoch. Deploy relay changes before their matching balancer changes as described above.
Migration requests are sent to PurrBalancer, which forwards them to the relay that owns the room. The relay treats a room incarnation as one game session:
- The client credential is the session membership credential. It stays stable across host claims so existing peers can reconnect, and rotates when an empty room name is reused.
- The host credential rotates after every successful claim and is a single-activation capability: only one host connection can consume it.
- Every room allocation or reuse gets a new opaque
incarnation; migrationgenerationstarts at zero for that incarnation. - A room waiting for a replacement host is reserved until that host connects or the empty-room timeout removes it, so allocation cannot replace an in-progress recovery.
- A claim starts one fixed 60-second activation lease, leaving margin beyond PurrNet's 30-second transition budget. Host socket authentication is provisional and does not reset the deadline; after PurrNet finishes server setup and local-player readiness it must call
/migration/activate. Ordinary clients that authenticate during the lease are not exposed to the provisional server, and their traffic is not forwarded, until activation. If readiness is not activated before the deadline, a peer can advance the generation and elect another host, atomically fencing the provisional socket and its expired credential. /migration/currentrequires the current client or host credential and returns metadata only. It never returns a credential.- A claim always requires the current client credential and exact
migration_claim_id,incarnation,expected_generation, andpromoted_player_idvalues.promoted_player_idmust be the canonical invariant decimal representation of a nonzero unsigned 64-bit PurrNet player ID (no sign, whitespace, or leading zeroes). The caller creates one opaque claim ID per migration attempt and reuses it for HTTP retries. - If the relay still has a live host, a claim also requires a one-time grant created by that host. After the relay observes host loss, the current client credential is enough for crash recovery.
/migration/current is a read-only GET. /migration/authorize, /migration/claim, and /migration/activate are POST operations. Request fields are carried in headers, and relay/balancer responses are globally marked Cache-Control: no-store, private with Pragma: no-cache so credentials and grants cannot be reused from an intermediary cache.
The balancer snapshots the room epoch, publication revision, and relay boot and forwards that identity in internal-only headers. Mutating migration calls hold only that room's route lease across the relay request; count publication, unregister, health removal, and relay replacement acquire the same room lease before a short endpoint gate. No monitor or relay-wide gate is held across the downstream request, so another room on the same relay remains concurrent. /migration/current uses an optimistic snapshot and validates it again after the read. The relay validates the forwarded epoch and boot before every read or mutation; a locally newer count revision is benign, but a future revision or changed owner is rejected.
If a mutating relay request times out or loses its response, the balancer returns retryable 503 migration_outcome_indeterminate. This means the relay may already have committed; it is not a claim that the operation rolled back or a proof of end-to-end strict linearizability. Callers must retry the exact authorization/claim/activation tuple or read current state. Exact claim and activation retries are idempotent, and an exact unexpired authorization retry returns the original grant and expiry.
Required headers: name, migration_secret.
The response contains relay connection fields plus roomName, incarnation, generation, fencingToken, promotedPlayerId, claimedAt, hostPhase, hostActive, claimPending, activationExpiresAt, and activatedAt. It contains neither secret nor the opaque claim ID: revealing either would let another holder of the shared client credential replay a winning claim and recover host authority.
hostPhase is relay-authoritative: pending means a claimed host is still inside its activation lease (whether or not its provisional socket has authenticated), active means the host explicitly confirmed full PurrNet readiness, expired means the lease elapsed, and vacant means no host owns the room. Callers should wait while it is pending, connect to the advertised host while it is active, and make a generation-checked claim only while it is expired or vacant. They do not need client-clock comparisons.
Required headers: name, host migration_secret, migration_claim_id, incarnation, expected_generation, and promoted_player_id.
The response contains grant, claimId, incarnation, generation, promotedPlayerId, and expiresAt. The grant is bound to the claim ID as well as the room state and promoted player. A newer authorization replaces the previous grant. Grants default to a 30-second lifetime; configure this with MIGRATION_GRANT_TIMEOUT_SECONDS on the relay.
Required headers: name, client migration_secret, migration_claim_id, incarnation, expected_generation, and promoted_player_id. Include migration_grant for a graceful handoff while the old host is connected.
A successful claim atomically consumes the grant, increments the generation, rotates the host credential and fencing token, records the promoted player, starts the fixed activation lease, and closes every old-generation relay socket. Closing all old sockets is intentional: even a peer that missed the application-level COMMIT deterministically observes host loss and enters recovery. The response includes the new host credential in secret, echoes claimId, and sets replayed to false. Before forwarding a claim success, the balancer validates its boolean TLS mode, credential, and usable relay ports, then injects the authoritative relay host; an incomplete upstream response becomes a sanitized retryable 502. The activation lease defaults to 60 seconds and can be configured with MIGRATION_HOST_ACTIVATION_TIMEOUT_SECONDS on the relay. That setting must exceed any customized PurrNet or HostMigration transition timeout, plus explicit HTTP, network, and scheduling margin. Shortening it tightens fencing but can reject a supported slow or retried promotion; lengthening it improves transient-failure tolerance but delays election after an abandoned attempt.
If that response is lost, repeating the exact claim ID, pre-claim incarnation and generation, and promoted player returns the same committed host credential and state with replayed: true while the fixed activation lease is still pending, or after that host has activated. Neither host authentication nor a claim replay extends the lease, advances the generation again, or disconnects a replacement host that has since connected. Once an unactivated lease expires, replay returns retryable host_activation_expired; the caller must read current state and make a fresh claim ID against the current generation. Reusing the same claim ID for a different tuple returns migration_claim_id_conflict.
Required headers: name, the claimed host migration_secret, migration_claim_id, incarnation, claimed expected_generation, promoted_player_id, and fencing_token.
The promoted host calls this only after PurrNet reports that the server role is connected and, for a listen host, its local player is ready. While the server is provisional, that listen-host client includes the claimed host credential as provisionalHostSecret alongside its normal client credential in socket authentication. The relay accepts only one such exact, unexpired local-client lane; all other clients remain isolated.
The relay verifies the exact current claim, the provisional host socket, and the unexpired lease under the same room/transport lock, atomically opens ordinary forwarding, and changes hostPhase from pending to active. Any ordinary client socket that raced into the pending lease is closed so it reconnects through the normal active-host path; this avoids relying on cross-socket packet ordering for its one-shot PurrNet login. The success response echoes claimId with the active room fence so the caller can validate the complete tuple. Repeating the exact activation is idempotent. A stale or superseded activation cannot revive an older generation. host_not_connected is retryable only before the one-use host credential is consumed; if that authenticated provisional socket is later lost, activation returns terminal host_connection_lost and the room retains its committed generation in vacant phase so peers can make a fresh claim.
Migration errors are JSON objects with code, error, and retryable fields and use an appropriate HTTP status instead of returning stack traces.
The client credential is shared session membership, not a per-player identity credential. Consequently, promoted_player_id is an assertion made by a trusted session member. Sessions that admit untrusted peers need a separate per-player credential (bound to the claimed player ID) before exposing migration claims to those peers.
PurrLay intentionally does not infer the PurrNet player roster from relay sockets: relay connections carry a shared session credential and are not authoritative stable-player membership. The promoted PurrNet host accepts the candidate-known roster as best-effort state, gives missing peers its normal migration grace period, and then retires the remaining ghosts through ordinary departure cleanup. An application coordinator may still confirm a departure or finalize the active roster earlier when it has better game-specific knowledge.