utils

@reaatech/agent-mesh-utils

Pluggable breaker persistence via BreakerStore + setBreakerStore — Firestore by default, plus InMemoryBreakerStore and the Postgres/Redis adapters.

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Per-agent circuit breaker implementation with Firestore persistence and cross-instance leader election. Prevents cascading failures by isolating unhealthy agents and provides distributed state synchronization across Cloud Run instances.

Installation

npm install @reaatech/agent-mesh-utils
# or
pnpm add @reaatech/agent-mesh-utils

Feature Overview

• Three-state circuit breaker — CLOSED (normal), OPEN (reject requests), HALF_OPEN (probing recovery)
• Exponential backoff — backoff multiplier doubles on each OPEN transition, up to 32×
• Leader-elected persistence — Firestore-based leader election ensures single-writer sync across instances
• Automatic state restoration — persisted circuit state survives Cloud Run restarts
• Configurable thresholds — failure count, reset timeout, half-open max calls — all environment-configurable

Quick Start

import { circuitBreaker } from "@reaatech/agent-mesh-utils";

// Check if an agent can receive traffic
if (circuitBreaker.canCall("serval")) {
  await dispatchToAgent(servalAgent);
}

// Record success or failure after dispatch
try {
  await mcpClient.sendMessage(context);
  circuitBreaker.recordSuccess("serval");
} catch (error) {
  circuitBreaker.recordFailure("serval");
}

API Reference

Circuit Breaker

circuitBreaker (singleton)

The global CircuitBreaker instance. All methods are synchronous and thread-safe.

Method	Description
getState(agentId)	Returns the current CircuitBreakerState, auto-transitioning if timeouts have elapsed
canCall(agentId)	Returns true if the circuit is CLOSED or HALF_OPEN with available slots
recordSuccess(agentId)	Records a successful call; closes the circuit if HALF_OPEN with enough successes
recordFailure(agentId)	Records a failure; opens the circuit if threshold reached
forceState(agentId, newState)	Forces the circuit to a specific state (testing/admin)
getAllStates()	Returns a snapshot of all circuit states
setState(state)	Sets a single circuit state (used for restoring from persistence)
setStates(states)	Sets multiple circuit states at once
clear()	Clears all circuit states (testing)

Circuit States

State	Behavior
CLOSED	Normal operation — requests pass through
OPEN	Failures >= threshold — requests rejected, auto-transitions to HALF_OPEN after RESET_TIMEOUT_MS * backoff_multiplier
HALF_OPEN	Testing recovery — limited test calls allowed, reverts to OPEN if any fail

Persistence Layer

startCircuitBreakerPersistence(): Promise<void>

Initializes leader election, restores states from Firestore, and starts periodic sync (leader only).

import { startCircuitBreakerPersistence, stopCircuitBreakerPersistence } from "@reaatech/agent-mesh-utils";

await startCircuitBreakerPersistence();

// On shutdown
stopCircuitBreakerPersistence();

stopCircuitBreakerPersistence(): void

Stops the sync interval and cleans up.

isLeader(): boolean

Returns true if this instance currently holds the leader lease.

getLeaderId(): string | null

Returns the current leader's instance ID.

restoreCircuitBreakerStates(maxRetries?): Promise<void>

Loads all circuit breaker states from Firestore with exponential-backoff retries.

updateCircuitBreakerState(state): Promise<void>

Updates local state and persists to Firestore.

getLocalCircuitBreakerState(agentId)

Returns the local circuit state for a given agent (no Firestore read).

Configuration

All thresholds are configured via environment variables (validated by @reaatech/agent-mesh):

Variable	Default	Description
CIRCUIT_BREAKER_FAILURE_THRESHOLD	5	Failures before opening circuit
CIRCUIT_BREAKER_RESET_TIMEOUT_MS	30000	Time before attempting recovery
CIRCUIT_BREAKER_HALF_OPEN_MAX_CALLS	3	Test calls allowed in HALF_OPEN
CIRCUIT_BREAKER_HALF_OPEN_TIMEOUT_MS	60000	Max time in HALF_OPEN before reverting to OPEN
CB_SYNC_INTERVAL_MS	5000	Leader sync interval
CB_LEADER_LEASE_MS	15000	Leader lease duration

Usage Patterns

With the Router

import { circuitBreaker } from "@reaatech/agent-mesh-utils";
import { env } from "@reaatech/agent-mesh";

async function dispatchToAgent(agent: AgentConfig) {
  if (env.ENABLE_CIRCUIT_BREAKER && !circuitBreaker.canCall(agent.agent_id)) {
    throw new Error(`Circuit breaker OPEN for agent ${agent.agent_id}`);
  }

  try {
    const response = await mcpClient.sendMessage(context);
    circuitBreaker.recordSuccess(agent.agent_id);
    return response;
  } catch (error) {
    circuitBreaker.recordFailure(agent.agent_id);
    throw error;
  }
}

Admin Operations

// Force-close a circuit (e.g., after fixing a downstream agent)
circuitBreaker.forceState("serval", "CLOSED");

// Inspect all circuits
for (const [agentId, state] of circuitBreaker.getAllStates()) {
  console.log(`${agentId}: ${state.state} (failures: ${state.failure_count})`);
}

Related Packages

• @reaatech/agent-mesh — Core types (CircuitBreakerState, CircuitState)
• @reaatech/agent-mesh-session — Firestore client (used by persistence layer)
• @reaatech/agent-mesh-router — MCP dispatch (uses circuit breaker for health gating)

License

MIT