refactor(iota-indexer): add fallback support for TransactionFilter{V2}::FromOrToAddress transaction query#11807
Conversation
…}::FromOrToAddress transaction query
iota-ci
added
infrastructure
kodemartin
left a comment
kodemartin
left a comment
There was a problem hiding this comment.
Regarding this
The indexer keeps no per-address metadata about which transactions have been pruned. When the database serves a FromOrToAddress query without a cursor in ascending order, it has no signal to distinguish what the initial transaction sequence number for this address was.
Can't we make a fallback call to resolve the earliest sequence number for the given address and start pagination there?
| ]; | ||
|
|
||
| let cursor_tx_seq = if let Some(cursor) = cursor { | ||
| let tx_seq = self.resolve_cursor_tx_digest_to_seq_num(cursor).await?; |
There was a problem hiding this comment.
This fails with a PostgresRead error
iota/crates/iota-indexer/src/read.rs
Lines 2744 to 2753 in 17b38ed
which implies that is_data_pruned || is_db_response_empty will return false at L1213 and the fallback will not be triggered.
So tx_digests yields the lower threshold in pagination. If its retention period is the same as tx_{sender,recipient} we effectively don't use the fallback.
| let cursor = match cursor { | ||
| Some(digest) => match self.cache_cursor.get(&digest) { | ||
| Some(tx_sequence_number) => Some(tx_sequence_number), | ||
| None => self.resolve_transaction_sequence_number(digest).await?, |
There was a problem hiding this comment.
If the transaction digest is not found this returns None which is the same as not providing a cursor. I reckon we should expect this to resolve into an Ok(Some. Otherwise, it is a user-input error no?
| "multi_fetch request failed with status: {}", | ||
| resp.status() | ||
| ))); | ||
| } | ||
|
|
||
| let bytes = resp.bytes().await?; | ||
| bcs::from_bytes::<Vec<T>>(&bytes).map_err(|e| { | ||
| IndexerError::Serde(format!("failed to deserialize multi_get response: {e:?}")) |
There was a problem hiding this comment.
multi_fetch and multi_get references are off here. Consider revising.
| /// storage through REST API interface. | ||
| client: HttpRestKVClient, | ||
| package_resolver: PackageResolver, | ||
| cache_cursor: MokaCache<TransactionDigest, TransactionSequenceNumber>, |
There was a problem hiding this comment.
cursor_cache would be more intuitive I reckon.
| limit: usize, | ||
| is_descending: bool, | ||
| options: iota_json_rpc_types::IotaTransactionBlockResponseOptions, | ||
| ) -> IndexerResult<Vec<IotaTransactionBlockResponse>> { |
There was a problem hiding this comment.
Couldn't we check the fallback.cache_cursor first to see quickly if we are in the pruned data range?
Co-authored-by: Konstantinos Demartinos <konstantinos.demartinos@iota.org>
…lter{V2}::FromOrToAddress transaction query
3 participants
Description of change
This PR adds historical fallback support to the indexer API for
TransactionFilter{V2}::FromOrToAddress.When no cursor is provided (limitation)
The indexer keeps no per-address metadata about which transactions have been pruned. When the database serves a
FromOrToAddressquery without a cursor in ascending order, it has no signal to distinguish what the initial transaction sequence number for this address was.Concretely: suppose an address had transactions in checkpoints 0 to 20 and the pruner has since dropped everything before checkpoint 21. A client now queries that same address in ascending order. The live SQL filter strips anything below the watermark, so the indexer returns the live data starting at checkpoint 21 moving forward until limit. The transactions from 0 to 20 are silently invisible, and nothing signals that the fallback should have been triggered.
This pruned data remains reachable by leveraging the cursor in descending order. If a client takes the oldest available live transaction digest (e.g., from checkpoint 21) and uses it as a cursor to query in descending order, the indexer will hit the
DataPrunederror , triggering the fallback to successfully retrieve the historical data from checkpoints 20 down to 0.When cursor is provided
The request cursor becomes the signal that triggers the fallback. When the cursor resolves to a
tx_sequence_numberbelow the watermark,ensure_data_not_pruned_for_txraises aDataPrunederror and the query is re-issued against the historical kv-store.We additionally fall back when the live query returns an empty page with no cursor, since that empty result is the only signal available for disambiguating "address has no history" from "the address's history is fully pruned".
Links to any relevant issues
fixes #11678
How the change has been tested
Basic tests (linting, compilation, formatting, unit/integration tests)
Patch-specific tests (correctness, functionality coverage)
Synced the indexer and the iota-kvstore worker upon a local netowrk
Issued
iotax_queryTransactionBlockswithFromOrToAddressfilter before pruning.Deleted records from
tx_senders&tx_recipients.Updated
min_available_txinwatermarkstable to reflect pruning.Issued the same queries to check if historical fallback would be triggered.
Pagination works as expected, only limitation is when no cursor is provided in ascending order. We hit the limitation described above.
Infrastructure QA (only required for crates that are maintained by @iotaledger/infrastructure)
Note
This patch does not affect the normal ingestion operation of the iota-indexer, thus no further tests were conducted.