Repository navigation
Observability & settlement-path hardening: alerting, health, metrics, auth, logging (C-091/110/111/112/113) - #242
Merged
Conversation
…e settlement path (C-091, C-110) Three M-tier backend items, all flag-gated and non-breaking. C-112 (#174) — operational alerting - lib/alerts.ts: sendAlert + formatAlert + 4 condition helpers (crank-failure, rpc-down, db-quota, dispute-spike). Opt-in via ALERT_WEBHOOK_URL (no-op when unset); never throws into a caller (a settlement path must not fail because alerting is down); pure formatting + injectable fetch -> unit-testable offline. - Wired live: crank-failure -> cron/finalize + HTTP finalize catches; dispute-spike -> both dispute routes (gated on ALERT_WEBHOOK_URL so the extra COUNT only runs when alerting is configured). - tests/unit/alerts.test.ts (11): format, no-op-when-unconfigured, posts-via-injected-fetch, swallow-errors, all 4 helpers. C-091 (#140) — auth on mutating routes - requireAuth(req) on the full create->accept->submit->cancel->finalize path + both dispute routes. No-op unless AUTH_ENFORCED is set. - finalize stays permissionless by design: auth is opt-in; under enforcement the frontend signs and a keeper bot uses an API key. C-110 (#170) — request correlation + tx-sig logging - log.forRequest(req) + on-chain tx sig logged on the same path (create_job / accept / submit / cancel / finalize / dispute). tsc clean; 316/316 unit tests; eslint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ettlement/dispute/fee/RPC metrics (C-111) Completes the M8 observability cluster on top of the alerting/logging commit: /api/health now reflects real subsystem status, and /api/metrics exposes the real settlement/dispute/fee/RPC figures. C-113 (#176) -- /api/health reflects real subsystem status - Added RPC reachability (Solana getSlot), on-chain program reachability (getAccountInfo(PROGRAM_ID) executable check -- reachability, not correctness), and crank liveness (overdue finalize backlog = Delivered jobs past challengeEndAt; tolerance via HEALTH_CRANK_BACKLOG_MAX). - 4s per-call timeout so a hung RPC can't stall the check; each failure degrades gracefully (rpc down -> program skipped; db down -> crank skipped). - Devnet smoke: rpc ok (slot=468110796), program ok (deployed+executable), crank/db degrade cleanly when DB is unreachable. C-111 (#172) -- /api/metrics reflects real on-chain activity - covenant_settlement_volume_usdc (sum of Finalized job amounts) - covenant_fees_accrued_usdc (settled volume * PROTOCOL_FEE_BPS -- derived from real settled amounts, 0 when 0) - covenant_disputes_total / _by_resolution / covenant_dispute_rate - covenant_rpc_up / _slot / _latency_ms (live getSlot probe) - Devnet smoke: rpc_up=1, slot/latency real; DB-derived gauges 0 with DB down (honest -- no fabricated settlement while the program is unredeployed, #236). tsc clean; 316/316 unit tests; eslint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…es + polling (C-115) (#243) Stacked on the observability PR (uses alertDbQuota from lib/alerts). Makes the Neon quota incident unable to recur silently, and trims the polling/query load that drives transfer cost. Alert before quota (the core) - lib/db-quota.ts: pure quotaUsageRatio + quotaLevel + env config, plus checkDbQuota (injectable size-probe + alert sink, never throws) and a pg_database_size probe. Opt-in via DB_QUOTA_LIMIT_BYTES (no-op when unset); warn/critical at DB_QUOTA_WARN_RATIO / DB_QUOTA_CRIT_RATIO (default 0.80/0.95). - app/api/cron/db-quota: CRON_SECRET-secured GET (same posture as the other crons) that probes size, always logs the level, and fires alertDbQuota at warn+. The cron cadence is the alert rate-limit (schedule it hourly). - tests/unit/db-quota.test.ts (17): ratio/level/config + every checkDbQuota path (no-op, under/at/over threshold, probe-failure, observer). Cut transfer cost - Polling: home (15s->30s) and settlement (10s->30s) data polls bumped to >=30s per the AC. (1s UI countdown/clock timers left alone — they hit no DB.) - Caching: /api/stats (home) wrapped in memoize (30s TTL) — collapses the 5 queries-per-poll into one set per window. (/api/settlement/stats already memoized.) - Index: @@index([updatedAt]) on Job — used by 10 hot "recent activity" orderBy queries and previously unindexed. (status / challengeEndAt / etc. already indexed.) Verified: prisma schema valid; tsc clean; 333/333 unit tests (+17); eslint clean; live smoke of the cron (no-auth / wrong-secret -> 401, Bearer -> 200 with the correct result JSON, probe-fail degrades to level=ok). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # app/app/api/jobs/route.ts
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Özet
M8 Observability & ops + M6 güvenlik kümesinden, on-chain blocker'a (#236)
bağımlı olmayan 5 backend issue'yu tek bir tutarlı **settlement-path gözlemlenebilirlik
hiçbir varsayılan davranış değişmiyor; her yeni davranış bir env değişkeniyle açılıyor.
Akış olarak bütünlük: log (C-110) → metric (C-111) → health (C-113) →
alert (C-112), üstüne auth (C-091).
2 commit · 10 dosya ·
tsctemiz · 316/316 unit test (+11) · eslint temiz · canlı devnet smoke.C-112 · Operasyonel alerting —
#174M8Ne: İnsan müdahalesi gereken koşullar için yapılandırılabilir bir webhook'a (Slack/
Discord/herhangi bir JSON endpoint) alert atan, opt-in ve asla throw etmeyen bir
alerting katmanı.
Nasıl / nerede:
lib/alerts.ts(yeni):sendAlert+formatAlert+ 4 koşul helper'ı(
alertCrankFailure,alertRpcDown,alertDbQuota,alertDisputeSpike).ALERT_WEBHOOK_URLset değilsesendAlertno-op (false döner, fetch çağrılmaz).bir settlement path'i alerting çöktü diye kırılmamalı.
fetchinject edilebilir → webhook olmadan unit-testlenebilir.alertCrankFailure→app/api/cron/finalizecatch'i ve HTTPapp/api/jobs/[id]/finalizecrank-failure catch'i (her ikisi fire-and-forgetvoid).alertDisputeSpike→ her iki dispute route'u (/api/disputes+/api/jobs/[id]/dispute); son 10 dk içindeki dispute sayısı eşiği aşarsa atar.tests/unit/alerts.test.ts(yeni, 11 test): format (severity/detail/fields, boşfield eleme), no-op-when-unconfigured, posts-via-injected-fetch, fetch-throw→false,
non-2xx→false, env'den okuma, 4 helper'ın her birinin doğru payload'u.
alertRpcDown(RPC failover'ın tümendpoint'leri tükettiği noktaya bağlamak, merge'lenmiş C-064
callWithFailover'ında"exhausted" ile "non-retryable"ı ayıran bir refactor ister — o merged kodu bozmadım) ve
alertDbQuota(doğru tetikleyicisi C-115'in dedup'lı db-size cron'u; metrics-scrape'ebağlamak her scrape'te spam üretirdi).
2'si tetikleyicisini bekleyen hazır helper.
ALERT_WEBHOOK_URLset'liyken çalışır — alerting kapalıykenekstra DB sorgusu yok (sıcak yolda sıfır overhead).
C-113 · Health checks —
#176M8Ne:
/api/health'in gerçek subsistem durumunu yansıtması (DB, RPC, program, crank).Nasıl / nerede (
app/api/health/route.ts):rpc—getServerConnection().getSlot()(Solana RPC erişilebilir mi).program—getAccountInfo(PROGRAM_ID); account var veexecutablemı.Kasıtlı olarak reachability kontrolü, correctness değil — deploy edilmiş ama buggy
bir program da "reachable" raporlar; "program reachable"ın dürüst anlamı bu.
crank— settlement backlog'u: cron finalizer'ın işlediği tam sorgu(
status=Delivered, challengeEndAt<=now). Backlog > tolerans ise crank ayakuyduramıyor demektir. Son
finalize_paymentTransaction zaman damgasını da raporlar.Tolerans
HEALTH_CRANK_BACKLOG_MAXile ayarlanır (varsayılan 0). Keyfi heartbeatyok — gerçek backlog sinyali.
Promise.race) → asılı bir RPC health'i kilitleyemez."skipped (db down)". Endpoint her zaman 200 + JSON döner;
oközetler.Doğrulandı (canlı devnet smoke, DB kasıtlı kapalı):
RPC + program kontrolleri gerçek devnet'e karşı çalışıyor; DB kopukken crank/db temiz
biçimde "skipped"e düşüyor.
C-091 · Mutating route'larda auth —
#140M6Ne: Değer/durum değiştiren route'larda doğrulanmış cüzdan imzası veya API key zorunluluğu.
Nasıl / nerede:
requireAuth(req)(merge'lenmişlib/require-auth—AUTH_ENFORCEDset değilseno-op) eklendi: tam para + dispute yolu —
POST /api/jobs(create) →accept→submit→cancel→finalize+POST /api/jobs/[id]/dispute. (/api/disputeszaten feat(backend): credit-market tests, mutating-endpoint auth (flag-gated), tx-sig logging #234'te wire'lıydı.)kendi
req.json()'ını sonra okumaya devam eder).finalizetasarımı gereği permissionless (challenge süresi sonrası herkessettle edebilir = sansür direnci). Auth opt-in olduğu için kırılmıyor;
AUTH_ENFORCEDaçılırsa frontend imzalar, keeper bot API key sunar — koda yorum olarak düştüm.
dokunulmadı.
hassas route'lar (profile/reviews/keys/referral/claims-list/agents-register) takip işi.
C-110 · Request correlation + on-chain tx-sig logging —
#170M8Ne: Her isteğin correlate edilebilmesi; on-chain tx sig'lerinin loglanması.
Nasıl / nerede:
log.forRequest(req)(route/method/request_id çıkaran, merge'lenmişlib/logger) +üretilen tx sig'in loglanması, C-091 ile aynı yola eklendi:
create_job/accept/submit/cancel/finalize_payment/dispute raised.finalize:reqLog.info("finalize_payment settled", { jobId, txHash }).(
log.forRequest'i zaten kullanan health/faucet/claims-buy/admin dışında) hâlâeklenmeyi bekliyor. "Tüm route'lar" daha geniş bir süpürme.
C-111 · Metrics —
#172M8Ne:
/api/metrics'te gerçek sayaçlar (settlement volume, dispute rate, fee accrual,RPC health).
Nasıl / nerede (
app/api/metrics/route.ts, mevcut Prometheus exposition'a eklendi):covenant_settlement_volume_usdc— Finalized job amount'larının toplamı (gerçeksettlement'ı yansıtır; sadece settle olanlar sayılır).
covenant_fees_accrued_usdc— settle olan hacim ×PROTOCOL_FEE_BPS. Gerçek settleedilmiş amount'lardan türetilir (uydurma değil); settlement 0 ise 0.
covenant_disputes_total+covenant_disputes_by_resolution{resolution}+covenant_dispute_rate(dispute/job oranı, 0..1).covenant_rpc_up+covenant_rpc_slot+covenant_rpc_latency_ms— canlıgetSlotprobe'u (4sn timeout).
Doğrulandı (canlı devnet smoke):
açıklamasındaki "wire a dashboard" (Grafana) bu PR'da yok — ayrı ops adımı. Ayrıca
#236 redeploy'a kadar gerçek on-chain settlement 0'dır; metrikler bunu dürüstçe
yansıtır (uydurma hacim yok).
Doğrulama
Bu PR'da olmayanlar (dürüst sınır)
alertDbQuota'yı dedup'lı bir db-sizecron'una bağlama + query caching/index + polling ≥30s. Ayrı PR (frontend + olası
migration + dedup'lı cron parçaları var).
alertRpcDown'ı RPC failover'ına canlı bağlama (C-064 refactor'u gerektirir).Closes #174
Closes #176
Refs #140 · Refs #170 · Refs #172