Overview
Enhance the ingestion workflow to support incremental data processing so that running ingestion does not reprocess the entire ledger history every time.
Acceptance Criteria
- Tracks last processed ledger: System persists the last successfully processed ledger. Upon subsequent ingestion runs, only new ledgers are processed.
- Idempotent execution: Running the ingestion multiple times without new data does not cause duplicate processing, errors, or side effects.
- CLI command available: A command-line interface is available to trigger incremental ingestion, e.g.,
python -m astroml.ingestion.incremental (or similar, per project conventions).
Notes
- Ensure the marker for the last processed ledger is robust and persists across restarts/crashes.
- Handle initialization (if no previous state is found) by starting from the configured start ledger or from the earliest available in the database.
- Update README with usage instructions for the new CLI.
Overview
Enhance the ingestion workflow to support incremental data processing so that running ingestion does not reprocess the entire ledger history every time.
Acceptance Criteria
python -m astroml.ingestion.incremental(or similar, per project conventions).Notes