fix: make deliveryRepository.popDueRetries atomic to close cross-replica retry race (#76) - #107
Merged
prodbycorne merged 12 commits intoJul 22, 2026
Conversation
The merge of SmartDropLabs#67 (fix-10-api-rate-limiting) into main pasted the entire top of the file twice and, within the second copy, declared `airdrops` twice in module.exports so the later object-literal key silently dropped expiryCheckIntervalSeconds/ledgerCacheTtlMs/expiryScanBatchSize used by the airdrop expiry job. Broke `require('./config')` outright (duplicate `const` in the same scope), taking every test that touches config with it.
Same bad-merge pattern as config.js: the whole file was pasted twice. Kept the newer body (health checks, price-source circuit state, airdrop expiry job wiring) and restored the API rate-limiting middleware mounts from the older copy, since that wiring is the actual feature SmartDropLabs#67 was merging in and would otherwise have been lost.
Same duplicated-file pattern; merged the two copies (keeping the PAYLOAD_TOO_LARGE coverage added by the newer copy) and fixed the RATE_LIMITED assertion to include retry_after_seconds, matching what src/middleware/rateLimit.js actually returns.
This assertion predates the airdrop expiry job's config additions (expiryCheckIntervalSeconds, ledgerCacheTtlMs, expiryScanBatchSize), which landed in a parallel PR and were never reflected here.
scanIds() was converted from SSCAN (plain set) to ZSCAN (sorted set) by SmartDropLabs#105, but this one assertion still checked mockRedis.sscan instead of mockRedis.zscan, so it silently asserted 0 > 1 was never reached via the real code path and would have passed for the wrong reason.
SmartDropLabs#105 converted apiKeys.js's id index from a plain Set (sadd/srem/ smembers) to a sorted Set (zadd/zrem/zrevrange) for stable pagination, but this test's inline Redis mock was never updated, so createKey() threw "redis.zadd is not a function" for every test that creates a key.
Same missing zadd/zrem/zrevrange/zcard/zscan gap as auth.test.js - airdrops.js's id index and expiry scan also moved to a sorted set under SmartDropLabs#105. Also removes leftover console.log debugging statements someone left in while chasing this same failure.
alerts.js's id index also moved to a sorted set under SmartDropLabs#105 (zadd/zrevrange/zcard/zrem); this route-level test's Redis mock only had smembers, so GET /api/v1/alerts 500'd before ever reaching the pagination assertions it's meant to check.
popDueRetries() was a ZRANGEBYSCORE followed by a separate ZREM - two round trips, not one atomic operation. Two backend replicas polling the shared webhooks:retries sorted set at the same time could both read the same due ids before either had removed them, double-claiming the same delivery and double-firing the webhook. Replaces it with a single Lua script (ZRANGEBYSCORE + ZREM in one round trip) registered via ioredis's defineCommand, so N concurrent callers against the same Redis always partition the due set instead of racing on it. cancelRetry/scheduleRetry/listByWebhook are untouched.
The atomic popDueRetries fix registers a custom Redis command via defineCommand, which the in-memory cache mock didn't implement. Adds a minimal emulation of the one custom command this codebase registers (popDueRetriesAtomic), reading and deleting without an intervening await so the mock preserves the same single-round-trip atomicity a real Lua script gets from Redis's single-threaded execution.
…pLabs#76) Covers the acceptance criteria from SmartDropLabs#76: two and four concurrent callers against a shared due-retry set never receive overlapping ids and the union exactly partitions what was seeded, plus happy-path/ empty-set/not-yet-due cases and a sanity check that cancelRetry, scheduleRetry, and listByWebhook still behave as before. Also includes a direct regression test that reproduces the original bug by calling the old read-then-delete-with-a-gap pattern directly against the mock Redis, proving the failure mode this fix closes rather than just asserting the new code happens not to trigger it.
…abs#76) Notes that popDueRetries' atomic Lua claim - not the worker's in-process `running` flag - is what makes it safe to run webhookRetryWorker on multiple backend replicas without double delivery, per the SmartDropLabs#76 acceptance criteria.
boluwacodes
force-pushed
the
fix/76-atomic-pop-due-retries
branch
from
July 22, 2026 12:43
53df22d to
83bcb0a
Compare
Contributor
|
All ci is green. well done |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #76.
popDueRetries()claimed due webhook retries from thewebhooks:retriesRedis sorted set via a
ZRANGEBYSCOREfollowed by a separateZREM-two round trips, not one atomic operation. Running more than one
backend replica (the normal case behind a load balancer) let two
instances read the same due ids before either had removed them,
double-claiming the same delivery and double-firing the webhook.
(
ZRANGEBYSCORE+ZREMin one round trip), registered once viaioredis'sdefineCommandso it's cached asEVALSHArather thanre-parsed every tick.
cancelRetry,scheduleRetry, andlistByWebhookare unchanged.top of
deliveryRepository.jsand in the README's retry-semanticssection (multi-replica safety comes from the atomic claim, not from
the worker's in-process
runningguard, which only prevents asingle process from overlapping with itself).
test/deliveryRepository.test.js: concurrent-caller testsproving no id is ever returned twice (2 and 4 concurrent callers
against a shared due set), plus happy-path/empty-set/not-yet-due
cases, an unchanged-behavior check for
cancelRetry/scheduleRetry/listByWebhook, and a direct reproduction of the originalread-then-delete race to prove this is closing a real failure mode.
test/helpers/cacheMock.js)with
defineCommandsupport so the mock preserves the samesingle-round-trip atomicity a real Lua script gets from Redis.
Unrelated CI breakage fixed along the way
main's CI has been red since PR #100 (4130067+): a chain of merges(#100, #105, #67) pasted large blocks of
src/config.js,src/index.js,and
test/errorHandler.test.jstwice, breakingrequire()outright, andleft several tests (
auth.test.js,airdrops.test.js,alerts-routes.test.js,airdrops-service.test.js,config.test.js)out of sync with the Redis Set → Sorted Set conversion from #105 (missing
zadd/zrem/zrevrange/zscanon inline test mocks, and one stalesscanassertion). Since this branch is built on top ofmain, none ofthat could be left broken and still show green CI here - each fix is its
own commit, clearly separated from the
#76work itself.Test plan
npm test- all 29 suites / 258 tests passnpx @redocly/cli lint openapi.yaml- passes (7 pre-existing warnings, no errors)src/config.jsandsrc/index.jsload withoutsyntax errors and export the fields other modules depend on
(
config.rateLimit,config.priceRateLimit,config.airdrops.*)