Description
Ingestor::process in crates/ingest/src/lib.rs always sets NewDeposit.operation_index = 0, regardless of which operation within a (potentially multi-operation) transaction the payment actually was. The (stellar_tx_hash, operation_index) unique index in crates/store/migrations/0001_init.sql was specifically designed as an anti-double-credit guard for exactly this case — two distinct payment operations inside the same transaction would both hash to (same_tx_hash, 0) and only the first would be recorded, silently dropping the second operation's credit. In practice horizon_op_id (the TOID, added in 0002_horizon_op_id.sql) already provides a globally-unique dedup key per operation and is the one actually enforced as unique today, so this is not causing double-credits, but the operation_index column is effectively storing wrong/misleading data that any future reporting or reconciliation query against it would trust incorrectly.
Requirements and Context
- Horizon's payment-record
id field is the operation TOID, which encodes (ledger_seq << 32 | tx_seq << 12 | op_index) per Horizon's TOID scheme — derive the real op_index by decoding the low 12 bits of the parsed id (parse it as a u64/i64 and mask/shift accordingly; confirm the exact bit layout against Horizon's documented TOID format before implementing, since getting the shift wrong would be worse than the current placeholder).
- Update
Ingestor::process in crates/ingest/src/lib.rs to compute and pass the real operation_index instead of the literal 0.
- This changes data for newly-ingested rows only — do not attempt to backfill existing rows in this PR; note in the PR description that a backfill (if desired) is a separate, deliberately-excluded follow-up given it touches production data.
Suggested Execution
Branch: fix/ingest/derive-real-operation-index
Implement Changes
- Add a small, pure
fn operation_index_from_toid(op_id: &str) -> Option<i32> helper (in crates/ingest/src/horizon.rs or lib.rs, wherever fits best alongside the existing PaymentRecord handling) implementing the TOID decode.
- Wire it into
Ingestor::process's NewDeposit construction, replacing the hardcoded 0.
Test and Commit
operation_index_from_toid_decodes_known_horizon_toid_examples (use real example TOIDs from Horizon's public documentation/API as fixtures, not invented values).
process_records_the_correct_operation_index_for_a_multi_op_transaction_toid in crates/ingest/tests/process_tests.rs.
- Run
cargo test -p octo-ingest locally before committing.
Example Commit Message
fix(ingest): derive the real operation_index from the Horizon TOID instead of hardcoding 0
operation_index was always recorded as 0, which is only harmless today
because horizon_op_id (the full TOID) is the index actually enforced as
unique for dedup. Derives the real per-operation index from the TOID's
encoded bits so the column reflects reality for any future reporting or
reconciliation that trusts it.
Guidelines
- Get the exact TOID bit layout right by cross-checking Horizon's own documentation/source before implementing — do not guess the shift amounts.
- Explicitly scope this PR to new rows only; do not write a backfill migration for existing
transactions rows as part of this change.
- Reference this issue with
Closes #<issue-number> in the PR description.
Description
Ingestor::processincrates/ingest/src/lib.rsalways setsNewDeposit.operation_index = 0, regardless of which operation within a (potentially multi-operation) transaction the payment actually was. The(stellar_tx_hash, operation_index)unique index incrates/store/migrations/0001_init.sqlwas specifically designed as an anti-double-credit guard for exactly this case — two distinct payment operations inside the same transaction would both hash to(same_tx_hash, 0)and only the first would be recorded, silently dropping the second operation's credit. In practicehorizon_op_id(the TOID, added in0002_horizon_op_id.sql) already provides a globally-unique dedup key per operation and is the one actually enforced as unique today, so this is not causing double-credits, but theoperation_indexcolumn is effectively storing wrong/misleading data that any future reporting or reconciliation query against it would trust incorrectly.Requirements and Context
idfield is the operation TOID, which encodes(ledger_seq << 32 | tx_seq << 12 | op_index)per Horizon's TOID scheme — derive the realop_indexby decoding the low 12 bits of the parsedid(parse it as au64/i64and mask/shift accordingly; confirm the exact bit layout against Horizon's documented TOID format before implementing, since getting the shift wrong would be worse than the current placeholder).Ingestor::processincrates/ingest/src/lib.rsto compute and pass the realoperation_indexinstead of the literal0.Suggested Execution
Branch:
fix/ingest/derive-real-operation-indexImplement Changes
fn operation_index_from_toid(op_id: &str) -> Option<i32>helper (incrates/ingest/src/horizon.rsorlib.rs, wherever fits best alongside the existingPaymentRecordhandling) implementing the TOID decode.Ingestor::process'sNewDepositconstruction, replacing the hardcoded0.Test and Commit
operation_index_from_toid_decodes_known_horizon_toid_examples(use real example TOIDs from Horizon's public documentation/API as fixtures, not invented values).process_records_the_correct_operation_index_for_a_multi_op_transaction_toidincrates/ingest/tests/process_tests.rs.cargo test -p octo-ingestlocally before committing.Example Commit Message
Guidelines
transactionsrows as part of this change.Closes #<issue-number>in the PR description.