Skip to content

Ensure checkpoint capture excludes non-started listeners, and null checkpoint data - #180

Merged
peterbroadhurst merged 1 commit into
mainfrom
checkpoint-loop-fixes
Sep 1, 2026
Merged

Ensure checkpoint capture excludes non-started listeners, and null checkpoint data#180
peterbroadhurst merged 1 commit into
mainfrom
checkpoint-loop-fixes

Conversation

@peterbroadhurst

@peterbroadhurst peterbroadhurst commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

The split in responsibilities between FFTM and FFCAPI plugins includes an eventstream, with multiple listeners.

Each of these listeners can have a separate checkpoint, as it can be in catchup mode behind the head group.

A checkpoint loop in FFTM runs regularly to persist a single checkpoint document for the whole eventstream.

The checkpoint loop rebuilds the full checkpoint document every cycle.

Problem

For a listener that has no checkpoint yet, it marshalled the nil in-memory checkpoint, writing a literal JSON null entry into the document.

On restart that null entry is treated as a real checkpoint and passed to the connector.
For an EVM block listener it unmarshals to a zero-valued checkpoint, so it restarts from block 0 instead of its configured fromBlock.

Note that normally the null never reaches the database, because in the same cycle the loop queries the connector (EventListenerHWM) and overwrites the entry with a real checkpoint.

There is a small possibility/window for one cycle of the loop to reach the database though - when that EventListenerHWM query can't answer.

One known edge case identified is a listener whose EventListenerAdd hasn't completed or has failed.

However, it's strange that the loop was querying anyway because listeners are added to the in-memory map before the connector accepts them.

Putting all these together, there's a small set of changes that tidy up this behavior:

Changes

  1. generateCheckpoint writes no entry for a listener with no checkpoint. Instead of a null entry. On restart the listener resolves its fromBlock fresh, which is the correct behavior for a listener that has never checkpointed.

  2. Listeners track a started flag, and only started listeners are queried for HWM checkpoints. started is set when the connector (or confirmation manager, for block listeners) accepts the listener — on EventListenerAdd success, stream start, or StartConfirmedBlockListener success — and cleared on stop. This stops the loop querying the connector for a listener it doesn't know about yet. A not-yet-started listener's existing in-memory checkpoint is still written to the document; only the query is gated.

  3. null entries in previously-persisted documents are ignored on load. Documents written by older versions can already contain null entries; buildAddRequest and buildBlockAddRequest now treat them as no checkpoint, so the listener starts from its configured fromBlock rather than a zero-valued checkpoint.

Checkpoints for listeners catching up from a historical fromBlock are unaffected: the connector's HWM during catchup is the listener's scan position, and it continues to be persisted every checkpoint interval.

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
@peterbroadhurst
peterbroadhurst requested a review from a team as a code owner August 28, 2026 19:00
}
// Add all started listeners with non-existent or stale checkpoints to the stale list,
// which we query below after we've dropped the lock
if l.started && (l.checkpoint == nil || l.lastCheckpoint == nil || time.Since(*l.lastCheckpoint.Time()) > es.checkpointInterval) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A not-yet-started listener's existing in-memory checkpoint is still written to the document; only the query is gated.

This behaviour looks reasonable to me. Suggest we add some logging here to help confirm whether a listener is not checking HWM because it's not started.

@peterbroadhurst
peterbroadhurst merged commit 3472b92 into main Sep 1, 2026
3 checks passed
@peterbroadhurst
peterbroadhurst deleted the checkpoint-loop-fixes branch September 1, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants