Failed connection establishment retries with no backoff and stampedes the whole pool (pathological against proxied/serverless Postgres)

[fabio-vapi]

Summary

When a connection fails while it is still being established (the socket closes cleanly mid-handshake, before ReadyForQuery), postgres.js retries it immediately, with no backoff — and every connection in the pool does this independently, so a burst that opened N connections retries all N in lockstep, forever. A single pool with max: 25 produces ~12,500 connection attempts per second, sustained, against an endpoint that is already failing to accept connections.

On traditional self-managed Postgres this is nearly invisible. On proxied / serverless Postgres — Neon, Aurora Serverless v2 (RDS Proxy), Supabase (Supavisor), etc. — where a connection-admission layer enforces hard attempt limits and sheds connections during establishment under load, it is exactly the wrong behavior: the client's own retry logic amplifies the very condition it is reacting to and turns a transient blip into a self-sustaining outage.

This report walks through the root cause, a reproduction (baseline + a deliberately catastrophic one), why the failure mode is becoming more common rather than less, and a proposed opt-in fix. A PR with tests is ready to follow.

How the pool reacts to a failed connection today

postgres.js has two distinct paths when a connection dies, and they behave very differently:

1. Error close — a socket error (RST / ECONNREFUSED / ECONNRESET), a connect timeout, or a protocol ErrorResponse. error() → errored() rejects the pending query and clears initial. The query fails; the connection does not loop. This is the path a classic FATAL: too many clients already (SQLSTATE 53300) takes — Postgres sends a proper error, and the caller sees it.

2. Clean close during establishment — the socket receives a FIN mid-handshake with no preceding error, so initial is still set when closed() runs. This is the path this issue is about, and it is the one a fronting proxy tends to produce when it drops a connection it isn't willing to admit.

Root cause

In src/connection.js, closed() returns early on the establishment path:

if (initial)
  return reconnect()

That return happens before the backoff bookkeeping a few lines below it:

closedTime = performance.now()
hadError && options.shared.retries++
delay = (typeof backoff === 'function' ? backoff(options.shared.retries) : backoff) * 1000

closedTime is initialised to 0 and is assigned only on that lower (post-drop) path. So on the establishment path reconnect() —

setTimeout(connect, closedTime ? Math.max(0, closedTime + delay - performance.now()) : 0)

— evaluates closedTime as falsy and schedules setTimeout(connect, 0): an immediate, tight retry loop. backoff never applies to establishment retries at all — it only ever governs reconnect-after-drop.

Two compounding problems:

• No backoff on establishment. A connection that can't be established retries as fast as the event loop will let it.
• No coordination across the pool. Each of the pool's max connection objects runs its own independent establishment retry, so they stampede in lockstep. There is nothing that says "one of us is already trying — the rest should wait."

Reproduction

Baseline

A fake endpoint that accepts TCP and then closes each connection cleanly mid-handshake (i.e. the establishment-failure path), counting inbound attempts:

import net from 'net'
import postgres from 'postgres'

let attempts = 0
const server = net.createServer(c => (attempts++, c.on('error', () => {}), c.on('data', () => c.end())))
server.listen(0, '127.0.0.1', () => {
  const sql = postgres({ host: '127.0.0.1', port: server.address().port, max: 5 })
  for (let i = 0; i < 5; i++) sql`select 1`.catch(() => {})
  setTimeout(() => { console.log('attempts/sec:', attempts); process.exit() }, 1000)
})
// observed: ~3,300 attempts in the first second, and it never stops

Catastrophic

The rate scales with pool size and never decays (no backoff), and in production it is multiplied by the number of application instances — each runs its own pool:

// same fake endpoint as above; vary max, or run several pools against one endpoint
for (const max of [5, 25, 50, 100]) { /* single pool */ }
// 8 independent pools of max:25 against one endpoint  => the aggregate a small fleet produces

Measured on a single machine (event-loop bound):

configuration	sustained connection attempts / sec
1 pool, max: 5	~3,300
1 pool, max: 25	~12,500
1 pool, max: 50	~17,800
1 pool, max: 100	~23,600

A single machine saturates its event loop around ~23k/s; a real deployment is N separate machines each contributing its own ~12k/s, so a modest fleet trivially puts hundreds of thousands of connection attempts per second onto an endpoint that is, by construction, already refusing them. Because there is no backoff, it does not subside on its own — the pool holds the endpoint down until traffic is cut or the process is killed.

Why this matters now (and increasingly)

Classic, self-managed Postgres rarely exercises this path: connection ceilings are hit the server answers with a protocol-level ErrorResponse (53300), which postgres.js routes down the error path (path 1 above) — the query rejects, no herd. Mid-handshake clean closes are unusual.

Proxied / serverless Postgres flips that. A connection-admission/pooler layer sits in front of the database and, under pressure, enforces per-source connection-attempt limits and drops connections during establishment rather than always returning a tidy protocol error. That is precisely the clean-close establishment path — and precisely where postgres.js currently has neither backoff nor coordination. Representative of the category:

• Neon — serverless Postgres behind a connection-admission proxy; over its limit it reports a generic XX000 "Too many connections attempts".
• Aurora Serverless v2 / RDS Proxy — connection multiplexing with borrow limits and throttling that can refuse or drop connections under contention.
• Supabase / Supavisor — a pooler layer with its own admission behavior.

As serverless and proxied Postgres become the default deployment model rather than the exception, this failure mode moves from "rare edge case" to "routine under load," and the current behavior means the client actively works against recovery. A fix here is forward-looking: it makes postgres.js a good citizen in front of exactly the endpoints it is increasingly deployed against.

Proposed solution

An opt-in reject_throttle option (default off; when off, behavior is identical to today). It fixes both problems above:

1. Backoff on establishment. Establishment retries use the existing backoff curve, instead of retrying immediately.
2. Coordinate the pool so the fleet stops stampeding:
   • On an establishment failure, the failed connection becomes the single prober and keeps retrying (with backoff); every other connection's open is blocked.
   • On each prober handshake success, an exponentially growing batch of the blocked connections is admitted (slow-start, 1 → 2 → 4 → … → max); a failed admission collapses the ramp back to 1.
   • Once the backlog drains, the breaker disengages back to baseline.
   • Liveness is the prober's own ongoing retry — no timers; if the prober goes away, the next admitted connection takes over.

This complements the existing backoff rather than duplicating it: backoff only delays each retry, and because its counter is on the shared pool object, every connection waits the same delay and then fires together — a delayed herd. reject_throttle ensures exactly one attempt is in flight during an outage and ramps capacity back up gracefully on recovery.

It deliberately governs only the clean-close establishment-reconnect path. Error-close (RST) and connect-timeout still reject exactly as before, so it never masks a hard failure or turns a rejection into an unbounded retry.

With it enabled, the reproduction above drops from ~12,500 attempts/sec to single digits, and recovers via slow-start the moment the endpoint starts accepting again.

PR

Implementation + a test suite (option plumbing, throttle-vs-herd, single-prober, slow-start recovery and disengagement, and the error-close / connect-timeout boundaries) is ready — I'll open it against this issue.