Ensure checkpoint capture excludes non-started listeners, and null checkpoint data - #180
Merged
Merged
Conversation
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Chengxuan
reviewed
Sep 1, 2026
| } | ||
| // Add all started listeners with non-existent or stale checkpoints to the stale list, | ||
| // which we query below after we've dropped the lock | ||
| if l.started && (l.checkpoint == nil || l.lastCheckpoint == nil || time.Since(*l.lastCheckpoint.Time()) > es.checkpointInterval) { |
Contributor
There was a problem hiding this comment.
A not-yet-started listener's existing in-memory checkpoint is still written to the document; only the query is gated.
This behaviour looks reasonable to me. Suggest we add some logging here to help confirm whether a listener is not checking HWM because it's not started.
Chengxuan
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The split in responsibilities between FFTM and FFCAPI plugins includes an eventstream, with multiple listeners.
Each of these listeners can have a separate checkpoint, as it can be in catchup mode behind the head group.
A checkpoint loop in FFTM runs regularly to persist a single checkpoint document for the whole eventstream.
The checkpoint loop rebuilds the full checkpoint document every cycle.
Problem
For a listener that has no checkpoint yet, it marshalled the nil in-memory checkpoint, writing a literal JSON
nullentry into the document.On restart that
nullentry is treated as a real checkpoint and passed to the connector.For an EVM block listener it unmarshals to a zero-valued checkpoint, so it restarts from block 0 instead of its configured
fromBlock.Note that normally the
nullnever reaches the database, because in the same cycle the loop queries the connector (EventListenerHWM) and overwrites the entry with a real checkpoint.There is a small possibility/window for one cycle of the loop to reach the database though - when that
EventListenerHWMquery can't answer.One known edge case identified is a listener whose
EventListenerAddhasn't completed or has failed.However, it's strange that the loop was querying anyway because listeners are added to the in-memory map before the connector accepts them.
Putting all these together, there's a small set of changes that tidy up this behavior:
Changes
generateCheckpointwrites no entry for a listener with no checkpoint. Instead of anullentry. On restart the listener resolves itsfromBlockfresh, which is the correct behavior for a listener that has never checkpointed.Listeners track a
startedflag, and only started listeners are queried for HWM checkpoints.startedis set when the connector (or confirmation manager, for block listeners) accepts the listener — onEventListenerAddsuccess, stream start, orStartConfirmedBlockListenersuccess — and cleared on stop. This stops the loop querying the connector for a listener it doesn't know about yet. A not-yet-started listener's existing in-memory checkpoint is still written to the document; only the query is gated.nullentries in previously-persisted documents are ignored on load. Documents written by older versions can already containnullentries;buildAddRequestandbuildBlockAddRequestnow treat them as no checkpoint, so the listener starts from its configuredfromBlockrather than a zero-valued checkpoint.Checkpoints for listeners catching up from a historical
fromBlockare unaffected: the connector's HWM during catchup is the listener's scan position, and it continues to be persisted every checkpoint interval.