Add NoteReader

[TomasArrachea]

Similar to the current AccountReader, it might be useful to have a NoteReader that would hold a reference to the store and implement methods to retrieve notes.

This would be useful in the context of the faucet, where we need to be able to retrieve notes while another task is generating transactions. The NoteReader would allow the client to give reading access to the notes to a separate task.

Alternatively, we could share the client behind a Mutex, but since generating transactions requires to lock the client, the access to read notes would be obstructed. Another potential solution could be to add interior mutability to the client so that generating transactions does not need to lock the whole client.

Context: 0xMiden/faucet#220

[Himess]

Can i take this ?

[bobbinth]

In the context of 0xMiden/protocol#2520, it would be great to prioritize this issue so that we can satisfy the requirements of the AggLayer integration work. One question for me is how much work implementing a flexible NoteReader would take and whether we can do it in stages so that we could satisfy the requirements of AggLayer sooner rather than later.

I think the first step is probably to define the interface and capabilities of the NoteReader. From the AggLayer perspective, we need the following:

• Ability to iterate over all input notes of a given account managed by the client.
• Ability to specify the block range for iteration.
• Ability to filter by note consumption status (e.g., only iterate over notes that have already been consumed). Though, if this info is available on per-note basis, filtering could be done separately.
• Making sure that iteration order follows the order in which the notes were executed (i.e., if note A was executed before note B, the iterator should return note A first, and then note B).

There are probably multiple ways to satisfy the above requirements - I don't have any opinions on this yet.

[igamigo]

One aspect that comes to mind with this is scale. For the AggLayer requirements, it seems there might be various different ways in which they may want to retrieve notes filtered from a very large set of input/output notes (but also, unsure of how much this will be true in practice). So it's probable that some amount of lazy loading/batching would be desired.

• Making sure that iteration order follows the order in which the notes were executed (i.e., if note A was executed before note B, the iterator should return note A first, and then note B).

The granularity for this is at the block level, right? I'm not sure there is anything more granular that would make sense to take into account, but double checking just in case.

[TomasArrachea]

• Ability to iterate over all input notes of a given account managed by the client.

Currently notes are not associated with accounts at a db level. The NoteScreener checks note consumability on the fly and it's used both during sync state and for the miden-client notes --list consumable command. We could use the same approach for the NoteReader: on each note fetch we could check consumability and filter by account id. Another possibility could be to track the consumability on the db, so it's only checked once on note import/sync. Not sure if that would have more complications.

Anyway, this would be useful for notes that are not yet consumed; once consumed/processing, the consumer id is part of the note state info. So, for starters we can just enable filtering by consumer id when checking for consumed notes.

---

I imagine the API for the reader could look something like this:

let note_reader = client
    .input_note_reader(status) // base status filter (Consumed, Committed, All, etc.)
    .for_consumer(account_id) // only notes consumable/consumed by this account
    .in_block_range(from, to) // filter by block range (nullifier block height for consumed notes, inclusion block for committed)
    .page_size(n) // internal page size for lazy fetching

let note_reader = client
    .output_note_reader(status)
    .for_sender(account_id) // only notes created by this account
    .in_block_range(from, to)
    .page_size(n);

And then it can be iterated with a .next() method that returns one note at a time, transparently fetching pages from the store at page_size notes. Results would be ordered by nullifier_block_num for consumed notes and inclusion_block_num for committed ones.

---

We would probably also benefit from some changes to the db model, since currently the notes tables hold the state/metadata serialized as a BLOB.

For output notes, we could add columns for sender_account_id, inclusion_block_num and nullifier_block_num to support SQL queries with this parameters. For input notes, the inclusion_block_num and nullifier_block_num columns, and potentially a consumer id, as mentioned above. These would have to be updated with the state changes.

[bobbinth]

Currently notes are not associated with accounts at a db level.

This is true for notes that haven't been consumed yet, but consumed notes should be associated with the account that consumed them, right?

The NoteScreener checks note consumability on the fly and it's used both during sync state and for the miden-client notes --list consumable command. We could use the same approach for the NoteReader: on each note fetch we could check consumability and filter by account id. Another possibility could be to track the consumability on the db, so it's only checked once on note import/sync. Not sure if that would have more complications.

This could work, but for the AggLayer use case, we are only interested in notes that have already been consumed. So, maybe there are two separate iterators - one over the notes that have been consumed (which should be pretty straight forward), and another one over notes that could be consumed by the account (we could do this as a future enhancement). The API the I described in 0xMiden/protocol#2520 (comment) assumes that we iterate over notes in the order in which they were consumed by the account.

I imagine the API for the reader could look something like this:

let note_reader = client
    .input_note_reader(status) // base status filter (Consumed, Committed, All, etc.)
    .for_consumer(account_id) // only notes consumable/consumed by this account
    .in_block_range(from, to) // filter by block range (nullifier block height for consumed notes, inclusion block for committed)
    .page_size(n) // internal page size for lazy fetching

let note_reader = client
    .output_note_reader(status)
    .for_sender(account_id) // only notes created by this account
    .in_block_range(from, to)
    .page_size(n);

I like these APIs. A couple of thoughts:

• As mentioned above, I'd start specifically with consumed input notes as this is the main use case and we can expand from there.
  • To support this use case properly, we may need to have a dedicated iterator. If so, it is probably fine to have a more narrow API.
• I probably wouldn't add page_size() - at least initially (unless it is trivial to support). Reading notes one-by-one from the DB should be fine initially.

Results would be ordered by nullifier_block_num for consumed notes and inclusion_block_num for committed ones.

It is very important to order consumed notes by their consumption sequence (not just by inclusion_block_num). Basically, we need to make sure that if note X was applied to an account before note Y, the iterator returns X before Y - even if both were consumed in the same block or even the same transaction. This may require adding a field do the database so that at insertion time we could assign "consumption order" to notes - but there could be other ways of doing this too.

For output notes, we could add columns for sender_account_id, inclusion_block_num and nullifier_block_num to support SQL queries with this parameters. For input notes, the inclusion_block_num and nullifier_block_num columns, and potentially a consumer id, as mentioned above. These would have to be updated with the state changes.

I would leave output notes (and unconsumed notes) for future improvements.

[igamigo]

Closing this issue as the first iteration was already merged (#1843). @TomasArrachea, could you open another one for future improvements/feature additions?