GetUTxO should provide ability to pass address as a filter

[matiwinnetou]

Why

When working with a large set of UTxOs / addresses responses from GetUTxO request can be substantial. In our case response is so large that it is clogging ability to issue further transactions.

What

Please provide ability to pass address as a filter, consider example:

{
   "tag":"GetUTxO".
   "address": "addr_test1qzlwg5c3mpr0cz5td0rvr5rvcgf02al05cqgd2wzv7pud6chpzk4elx4jh2f7xtftjrdxddr88wg6sfszu8r3gktpjtqrr00q9"
}

should only return UTxOs which are matching this address.

How

While generating response for GetUTxO simply filter out any other address.

[abailly-iohk]

@matiwinnetou Seems like an interesting idea to explore, could you please tell us a bit more about the why? I could imagine a more general querying/filtering mechanism could be useful but we would like to better understand the need behind the want. Happy to have a chat together to refine my understanding.

[matiwinnetou]

We are trying to process 1 mln votes on Hydra, first we import votes and and then we process them. Not only we need to narrow it down by address by also paginate through it. There is no Hydra indexer or anything like this so we need ability to be able to query the hydra node in an effective way to construct transactions. Right now what we have is hardly scalable for anything of the magnitude we are working on.

[abailly-iohk]

Here is the picture of Hydra Voting's situation:

• The client is currently using QueryUTxO and the filtering is done on the client side. This is not a big deal right now but will be cumbersome when there's a lot of votes to process, whether those votes are distributed over a large UTxO set or inside transactions with bigger limits.
• We discussed how the client could also benefit from being more "reactive", eg. use the SnapshotConfirmed or other events emitted by the node to decide what to process
• This is still problematic in terms of amount of data transferred because, again, of the size of the current snapshot in the face of millions of votes
• The query/response pattern with filtering capabilities over some REST interface (à la Blockfrost) makes it easier to build stateless clients and fits well the web paradigm
• Alternative (or complementary way) would be to provide filtering capabilities over the stream of ServerOutput, à la GraphQL Subscriptions

[ch1bo]

GraphQL sounds nice, but I would prefer to keep the backing data model separate from the internal state available in the hydra-node right now. I wonder whether it would make sense to take our current API events and stream them into a separate component, which then provides the graphql API.

[matiwinnetou]

We need filtering for both SnapshotConfirmed and filter requests to GetUTxO (e.g. pass address / pagination support / e.g retrieve only first 100 UTxOs).

• For SnapshotConfirmed some language query language for event streaming can be used, e.g. ksql (there are many many alternatives)
• We don't strictly need GetUTxO, Hydra can also provide REST interface somewhat compatible with Blockfrost (at least in terms of transaction building) so we don't have to each time download SnapshotConfirmed.

We did a test and SnapshotConfirmed completely killed our application on larger data sets and on connecting back to the cluster it was close to insane.

SnapshotConfirmed is needed for archival nodes but this is a bit esoteric use case, where there could be a special hydra client within a network which would be storing and indexing all snapshots. Even then on re-connect one needs to start from previous seq because otherwise archival node would reprocess all snapshots since beginning of the blockchain. This is not really necessary because of course this archival node already has this data. We could compare this for cardano-node to boot up and always start from the genesis block.

I am aware there is a separate workstream regarding ditching partially history on re-connect. I am not sure exactly how you plan to implement this, if would also be done with some event query API to be honest, as an example:

• archival node would ask to give all snapshots while it was off-line
• our use case would simply ask only for the latest SnapshotConfirmed(!) or in case or reconnect only use GetUTxO / REST API in Hydra node.

[abailly-iohk]

@matiwinnetou We discussed this issue within the team and came to the conclusion we need to propose a proper Query interface to the node to properly address your and possibly others' needs. As it stand theGetUTxO message is clearly a kludge that goes against the "grain" of the current asynchronous duplex API exposed by Hydra node (see ADR-3) which is meant to be fully message oriented.

From a high-level architecture, the existing client API fulfils the Command Responsibility of the CQRS pattern, and also provides an Event stream exposing the changes in the internal state of the hydra node, but is not fit for querying. In general, this is inspired by how applications interact with the cardano-node, the latter exposing similar duplex asynchronous protocols and leaving the work of maintaining state to client applications or thirdparty tools (eg. indexers, db-sync...).

We have started investigating what such a Query API would look like as there are a few options we can choose from: Should it be synchronous REST-like? Or should we propose a GraphQL API including both queries and subscriptions? What would be the extent of state exposed by this API?

But designing and implementing this API could take some time as we would like to involve the small but active community of people involved in building stuff on Hydra. To address your concerns in the short-term, we would like to propose you the following plan:

• We deprecate GetUTxO message to clearly notify users this should not be relied on to build apps,
• We provide a way to trim down the size of SnapshotConfirmed, passing only the newly confirmed transactions,
• This implies the client (you) will be responsible for maintaining a state (eg. UTxO map) updated from the transactions confirmed. We will therefore adapt the code of Hydraw, our demo application, to that pattern,
• We would be happy to spend some time with you to work on adapting existing code.

[uhbif19]

Event stream exposing the changes in the internal state of the hydra node, but is not fit for querying.

Why? General type client will have WS connection anyway, so supporting WS gonna be just another code responsibility.

Or should we propose a GraphQL API

Would GraphQL API cover complex queries? Also I think GraphQL server almost requires library/server usage, which may be of varying quality on different PLs.

I think querying/filtering UTxOs is some common case for L2 and indexing solutions. Maybe IOG could create some common standard on that?

[ch1bo]

A solution outlined and also pursued in the past by some users of Hydra was to feed the transaction information to Ogmios/Kupo.

@Quantumplation and his team had adapted Ogmios for that purpose and may consider open-sourcing their work (working title: HOgmios). I suppose a direct integration of feeding events into Kupo directly would also be possible.

End result of this, would be means to aggregate and query UTxOs as indexed by kupo in a REST-like API, which may work with this big volume of UTxOs .