Skip to content

Allow incremental ingestion without reprocessing entire history #35

@hotoke-no-Kami

Description

@hotoke-no-Kami

Overview

Enhance the ingestion workflow to support incremental data processing so that running ingestion does not reprocess the entire ledger history every time.

Acceptance Criteria

  • Tracks last processed ledger: System persists the last successfully processed ledger. Upon subsequent ingestion runs, only new ledgers are processed.
  • Idempotent execution: Running the ingestion multiple times without new data does not cause duplicate processing, errors, or side effects.
  • CLI command available: A command-line interface is available to trigger incremental ingestion, e.g., python -m astroml.ingestion.incremental (or similar, per project conventions).

Notes

  • Ensure the marker for the last processed ledger is robust and persists across restarts/crashes.
  • Handle initialization (if no previous state is found) by starting from the configured start ledger or from the earliest available in the database.
  • Update README with usage instructions for the new CLI.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions