Skip to content

fix(ingest): Stop hardcoding operation_index to 0 and derive it from the Horizon operation TOID #94

Description

@Emmyt24

Description

Ingestor::process in crates/ingest/src/lib.rs always sets NewDeposit.operation_index = 0, regardless of which operation within a (potentially multi-operation) transaction the payment actually was. The (stellar_tx_hash, operation_index) unique index in crates/store/migrations/0001_init.sql was specifically designed as an anti-double-credit guard for exactly this case — two distinct payment operations inside the same transaction would both hash to (same_tx_hash, 0) and only the first would be recorded, silently dropping the second operation's credit. In practice horizon_op_id (the TOID, added in 0002_horizon_op_id.sql) already provides a globally-unique dedup key per operation and is the one actually enforced as unique today, so this is not causing double-credits, but the operation_index column is effectively storing wrong/misleading data that any future reporting or reconciliation query against it would trust incorrectly.

Requirements and Context

  • Horizon's payment-record id field is the operation TOID, which encodes (ledger_seq << 32 | tx_seq << 12 | op_index) per Horizon's TOID scheme — derive the real op_index by decoding the low 12 bits of the parsed id (parse it as a u64/i64 and mask/shift accordingly; confirm the exact bit layout against Horizon's documented TOID format before implementing, since getting the shift wrong would be worse than the current placeholder).
  • Update Ingestor::process in crates/ingest/src/lib.rs to compute and pass the real operation_index instead of the literal 0.
  • This changes data for newly-ingested rows only — do not attempt to backfill existing rows in this PR; note in the PR description that a backfill (if desired) is a separate, deliberately-excluded follow-up given it touches production data.

Suggested Execution

Branch: fix/ingest/derive-real-operation-index

Implement Changes

  • Add a small, pure fn operation_index_from_toid(op_id: &str) -> Option<i32> helper (in crates/ingest/src/horizon.rs or lib.rs, wherever fits best alongside the existing PaymentRecord handling) implementing the TOID decode.
  • Wire it into Ingestor::process's NewDeposit construction, replacing the hardcoded 0.

Test and Commit

  • operation_index_from_toid_decodes_known_horizon_toid_examples (use real example TOIDs from Horizon's public documentation/API as fixtures, not invented values).
  • process_records_the_correct_operation_index_for_a_multi_op_transaction_toid in crates/ingest/tests/process_tests.rs.
  • Run cargo test -p octo-ingest locally before committing.

Example Commit Message

fix(ingest): derive the real operation_index from the Horizon TOID instead of hardcoding 0

operation_index was always recorded as 0, which is only harmless today
because horizon_op_id (the full TOID) is the index actually enforced as
unique for dedup. Derives the real per-operation index from the TOID's
encoded bits so the column reflects reality for any future reporting or
reconciliation that trusts it.

Guidelines

  • Get the exact TOID bit layout right by cross-checking Horizon's own documentation/source before implementing — do not guess the shift amounts.
  • Explicitly scope this PR to new rows only; do not write a backfill migration for existing transactions rows as part of this change.
  • Reference this issue with Closes #<issue-number> in the PR description.

Metadata

Metadata

Assignees

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial Campaign | FWC26Campaign: Official Campaign | FWC26area/backendBackend crates: api, store, ingest, webhooks, bin/serverdifficulty/mediumMedium difficulty

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions