Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ucanto server onInvocationHandled callback #169

Open
vasco-santos opened this issue Dec 8, 2022 · 4 comments
Open

ucanto server onInvocationHandled callback #169

vasco-santos opened this issue Dec 8, 2022 · 4 comments

Comments

@vasco-santos
Copy link
Member

We are implementing store/* + upload/* in https://github.com/web3-storage/upload-api/ and we want to perform asynchronous post processing on ucan invocations handled by the upload-api service.

We currently need to decode the CBOR in the end of the invocation handling to properly add it to our processing system.

const bytes = Buffer.from(request.body, 'base64')
const car = await CAR.codec.decode(bytes)
const root = car.roots[0].cid.toString()
const cbor = CBOR.codec.decode(car.roots[0].bytes)

Having ucanto-server to support something like onInvocationHandled callback in Server.create or emitting an Event would be great to pass the handled CBOR invocation. Thoughts about this @Gozala ?

@Gozala
Copy link
Collaborator

Gozala commented Dec 10, 2022

provide(capability, handler) used in servers is a decorator that simply does the validation, you could simply decorate it with another function that does something with the the passed invocation + context

* @param {API.Invocation<API.Capability<A, R, API.InferCaveats<C>>>} invocation
* @param {API.InvocationContext} options
*/
async (invocation, options) => {
const authorization = await access(invocation, {
...options,
capability,
})
if (authorization.error) {
return authorization
} else {
return /** @type {API.Result<Exclude<U, {error:true}>, {error:true} & Exclude<U, Exclude<U, {error:true}>>|API.InvocationError>} */ (
handler({
capability: authorization.capability,
invocation,
context: options,
})
)
}

It is also worth pointing out that Invocation passed is just a delegation with single capability

export interface Invocation<C extends Capability = Capability>
extends Delegation<[C]> {}

Which in turn has both cid bytes and data fields that correspond to the cid, block bytes and decoded block cbor (assuming it was in CBOR, it could be JWT as well).

export interface Delegation<C extends Capabilities = Capabilities> {
readonly root: UCANBlock<C>
/**
* Map of all the IPLD blocks that were included with this delegation DAG.
* Usually this would be blocks corresponding to proofs, however it may
* also contain other blocks e.g. things that `capabilities` or `facts` may
* link.
* It is not guaranteed to include all the blocks of this DAG, as it represents
* a partial DAG of the delegation desired for transporting.
*
* Also note that map may contain blocks that are not part of this
* delegation DAG. That is because `Delegation` is usually constructed as
* view / selection over the CAR which may contain bunch of other blocks.
*/
readonly blocks: Map<string, Block>
readonly cid: UCANLink<C>
readonly bytes: ByteView<UCAN.UCAN<C>>
readonly data: UCAN.View<C>

So you should be able to store invocation block either in the handler or use a decorator to do it before or after handler runs. Unfortunately we do not retain the CAR CID we received the invocation from, perhaps we could add it as metadata to invocation / delegation if it's needed.

@Gozala
Copy link
Collaborator

Gozala commented Dec 10, 2022

Also please note that assumption here is incorrect

const bytes = Buffer.from(request.body, 'base64')
const car = await CAR.codec.decode(bytes)
const root = car.roots[0].cid.toString()
const cbor = CBOR.codec.decode(car.roots[0].bytes)

Above seems to assume there will be a single invocation in the CAR, but in practice it could be multiple each corresponding to won root.

I also don't think body should be base64 encoded string (unless it's some other AWS quirk)

@Gozala
Copy link
Collaborator

Gozala commented Dec 10, 2022

@vasco-santos from our conversation I was under impression that ideally we'd just add toJSON method to delegation so you could get a JSON corresponding to the invocation block. delegation.bytes already give you block bytes and delegation.cid give you the CID of the invocation.

Would that be sufficient or do we need more than that ?

@vasco-santos
Copy link
Member Author

My initial goal was to make it in a way that all handlers could benefit from this out of the box. However, we are now seeing a use case that makes us need to rethink this. So, adding a wrapper on the handler will actually need to be the way to go.

I also don't think body should be base64 encoded string (unless it's some other AWS quirk)

Yeah, is AWS quirk.

Above seems to assume there will be a single invocation in the CAR,

Oh yes, this is good to know!

@vasco-santos from our conversation I was under impression that ideally we'd just add toJSON method to delegation so you could get a JSON corresponding to the invocation block. delegation.bytes already give you block bytes and delegation.cid give you the CID of the invocation.
Would that be sufficient or do we need more than that ?

I think this is enough, but I still want to make sure if we want the actual CAR cid. I will sync with Oli on this too before

vasco-santos added a commit to storacha-network/w3infra that referenced this issue Dec 13, 2022
Adds kinesis ucan log stream to ucanto service. Once a UCAN invocation
is handled by the service, it is sent to Amazon Kinesis data streams for
post processing (JSON with invocation CID, invocation bytes, and decoded
invocation).

Kinesis log stream has its own stack named `UcanStreamStack` which will
include resources needed for post processing of ucan stream ops.
`ApiStack` depends on `UcanStreamStack` given it will use its stream, as
well as its data further down the line to get content like user facing
stats

Per https://www.notion.so/UCAN-LOG-0f3870fc4b404f5cbf646bf16b463365

Implementation details:
- Invocation view content
- `{ carCid: string, value: { att: UCAN.Capabilities, aud:
'did:${string}:${string}', iss: 'did:${string}:${string}' } }`
- having att, audience and issuer should be enough for all the
operations we intend to perform. Skipped `prf`, `exp`, `nbf`, `fct`,
`nnc`, `v`, and `signature`.
  - see format in comment below

Other notes:
- SST Kinesis guide:
https://sst.dev/examples/how-to-use-kinesis-data-streams-in-your-serverless-app.html
- we are configuring 365 days for now
https://docs.aws.amazon.com/cdk/api/v1/docs/aws-kinesis-readme.html#streams
- aws related deps updated to use same everywhere
- we are currently doing redundant encodings/decodings that could be
avoided if we do web3-storage/ucanto#169
- Follow up PRs will be created with consumers:
  - data aggregation lambda for user facing stats
  - dynamoDB
  - ...
- NOTE: they can start from before as we talked
https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-lambda-event-sources.KinesisEventSourceProps.html#properties
- we need to define a partition key, but we do not need to commit now to
one. Only by the time we want more shards for scaling up

Follow ups:
- #98

Co-authored-by: Alan Shaw <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants