Skip to content

feat: add idempotent ledger checkpoint system - #650

Merged
Chucks1093 merged 3 commits into
accesslayerorg:mainfrom
wheval:feat/ledger-checkpoint-system
Jul 27, 2026
Merged

feat: add idempotent ledger checkpoint system#650
Chucks1093 merged 3 commits into
accesslayerorg:mainfrom
wheval:feat/ledger-checkpoint-system

Conversation

@wheval

@wheval wheval commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

📌 Summary

Adds a ledger checkpoint system to prevent duplicate event processing after indexer restarts.

🎯 Purpose / Motivation

When the indexer restarts mid-batch, it currently has no reliable mechanism to resume from the exact ledger and event offset. This causes duplicate trade records and gaps in trade history. A checkpoint system makes restart recovery deterministic and safe.

🛠️ Changes Made

  • Add idempotent ledger checkpoint system to prevent duplicate event processing after indexer restart #632: Add idempotent ledger checkpoint system
  • Added LedgerCheckpoint Prisma model with ledger, lastProcessedEventIndex, batchHash, completedAt
  • Created checkpoint service with write, read, validate, and resume functions
  • Non-blocking checkpoint writes with 5s timeout (falls back to logging on failure)
  • Batch hash mismatch detection triggers warn log and full ledger reprocess
  • Integrated checkpointing into event processor with ledger-grouped processing
  • Added unit and integration tests for checkpoint operations

🧪 How to Test

  1. Run pnpm test to verify all tests pass
  2. Check that LedgerCheckpoint model is in Prisma schema
  3. Verify checkpoint service writes/reads correctly via integration tests
  4. Verify event processor groups events by ledger and checkpoints after each

⚠️ Breaking Changes

  • None.

🔗 Related Issues

Closes #632

✅ Checklist

  • Code builds successfully
  • Tests added/updated
  • No console errors
  • Documentation updated (if needed)

wheval added 3 commits July 25, 2026 00:05
- Add local setup guide with prerequisites, step-by-step instructions, and troubleshooting
- Add API endpoint reference with all public endpoints, auth requirements, and sample responses
- Cross-link both documents for easy navigation

Closes accesslayerorg#598
- Add LedgerCheckpoint Prisma model with ledger, lastProcessedEventIndex, batchHash, completedAt
- Create checkpoint service with write, read, validate, and resume functions
- Non-blocking checkpoint writes with 5s timeout
- Batch hash mismatch detection triggers warn log and reprocess
- Integrate checkpointing into event processor with ledger-grouped processing
- Add unit and integration tests for checkpoint operations

Closes accesslayerorg#632
@Chucks1093
Chucks1093 merged commit c9bdefe into accesslayerorg:main Jul 27, 2026
1 check passed
@Chucks1093

Chucks1093 commented Jul 27, 2026

Copy link
Copy Markdown
Member

Nice work on this. The checkpoint system is a clean solution. Grouping events by ledger and writing the checkpoint after each batch makes restarts safe without any complex state to manage. The 5s timeout on checkpoint writes is a good call too, it keeps the main flow from stalling if the write is slow. Good test coverage as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add idempotent ledger checkpoint system to prevent duplicate event processing after indexer restart

2 participants