feat(indexer): add structured batch start/completion logs - #658
Merged
Conversation
Add per-batch structured logging to processIndexerChainEvents so operators can see indexer throughput and batch size without querying the database. - Emit an info-level "indexer_batch_started" log before processing, with from_ledger, to_ledger (min/max ledger across the batch), and batch_size (raw event count before dedup). - Emit a debug-level "indexer_batch_completed" log after processing, with from_ledger, to_ledger, events_processed (unique count after dedup), and duration_ms (wall-clock time via the existing monotonic clock helper). - Both logs fire exactly once per batch call, not per individual ledger/event; the existing per-event log is untouched. Closes accesslayerorg#648
|
@KingDavid9999 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
processIndexerChainEventsinsrc/utils/indexer-event-processor.utils.ts, the shared function that processes an incoming batch of chain events (currently only called byprocessTradeEvents).indexer_batch_started) before processing starts, withfrom_ledger,to_ledger(min/maxledgeracross the batch), andbatch_size(raw event count received, before dedup).indexer_batch_completed) after processing finishes, withfrom_ledger,to_ledger,events_processed(unique event count after dedup), andduration_ms(measured with the same monotonic clock helper already used for per-event timing).docs/indexer/EVENT_PROCESSING.mdwith a new section documenting the two log shapes and field meanings.Note on
batch_sizesemantics: this server only receivesevents: ChainEvent[]— it has no visibility into the ledger range the external polling worker actually requested (seePIPELINE_REFERENCE.md). Sofrom_ledger/to_ledger/batch_sizeare derived from the ledgers present on the events in the batch, not from an externally-supplied range. Flagging this in case the intended semantics were "ledger count in the requested range" rather than "event count in this call" — happy to add explicit range params if so.Testing
pnpm lintpnpm buildpnpm exec prisma generate— no schema changespnpm test -- indexer-event-processor.utils.test.ts— 8/8 passing, including 4 new tests for batch-start fields, batch-completion fields, single-log-per-batch, and the no-ledger edge casepnpm test -- indexer-pipeline.integration.test.ts— unaffected, 3/3 passingChecklist
EVENT_PROCESSING.md)Closes #648