feat: add kinesis log stream

[vasco-santos]

Adds kinesis ucan log stream to ucanto service. Once a UCAN invocation is handled by the service, it is sent to Amazon Kinesis data streams for post processing (JSON with invocation CID, invocation bytes, and decoded invocation).

Kinesis log stream has its own stack named UcanStreamStack which will include resources needed for post processing of ucan stream ops. ApiStack depends on UcanStreamStack given it will use its stream, as well as its data further down the line to get content like user facing stats

Per https://www.notion.so/UCAN-LOG-0f3870fc4b404f5cbf646bf16b463365

Implementation details:

• Invocation view content
  • { carCid: string, value: { att: UCAN.Capabilities, aud: 'did:${string}:${string}', iss: 'did:${string}:${string}' } }
  • having att, audience and issuer should be enough for all the operations we intend to perform. Skipped prf, exp, nbf, fct, nnc, v, and signature.
  • see format in comment below

Other notes:

• SST Kinesis guide: https://sst.dev/examples/how-to-use-kinesis-data-streams-in-your-serverless-app.html
  • we are configuring 365 days for now https://docs.aws.amazon.com/cdk/api/v1/docs/aws-kinesis-readme.html#streams
• aws related deps updated to use same everywhere
• we are currently doing redundant encodings/decodings that could be avoided if we do ucanto server onInvocationHandled callback ucanto#169
• Follow up PRs will be created with consumers:
  • data aggregation lambda for user facing stats
  • dynamoDB
  • ...
  • NOTE: they can start from before as we talked https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-lambda-event-sources.KinesisEventSourceProps.html#properties
• we need to define a partition key, but we do not need to commit now to one. Only by the time we want more shards for scaling up

Follow ups:

• Kinesis events with verifiable TS #98

Needs:

• feat: persist ucan invocation car to s3 and replicate to r2 #83

[seed-deploy]

View stack outputs

• pr86-upload-api-ApiStack

  Name	Value
  ApiEndpoint	https://toyiez9aab.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr86.up.web3.storage

• pr86-upload-api-BusStack

• pr86-upload-api-CarparkStack

• pr86-upload-api-SatnavStack

• pr86-upload-api-UcanInvocationStack

[vasco-santos]

EDIT: SEE COMMENTS LATER AS THIS WAS MODIFIED

Data written into kinesis stream

{
    "carCid": "bafyreibr5w4fjaxg5da5gwovujsz6vmxp4dwtsm73iiiwki7ym5gon7p5q",
    "data": {
        "att": [
            {
                "nb": {
                    "link": {
                        "code": 514,
                        "version": 1,
                        "hash": {
                            "0": 18,
                            "1": 32,
                            "2": 132,
                            "3": 245,
                            "4": 3,
                            "5": 183,
                            "6": 72,
                            "7": 151,
                            "8": 231,
                            "9": 183,
                            "10": 40,
                            "11": 33,
                            "12": 244,
                            "13": 229,
                            "14": 22,
                            "15": 202,
                            "16": 14,
                            "17": 50,
                            "18": 153,
                            "19": 111,
                            "20": 251,
                            "21": 187,
                            "22": 157,
                            "23": 131,
                            "24": 194,
                            "25": 123,
                            "26": 227,
                            "27": 195,
                            "28": 143,
                            "29": 11,
                            "30": 74,
                            "31": 2,
                            "32": 43,
                            "33": 98
                        }
                    },
                    "size": 225
                 },
            "can": "store/add",
            "with": "did:key:z6Mkf4Tc5v4LWessC2cExyQNjprsxJPGfv82pvNmzUXchdPM"
            }
        ],
        "aud": "did:key:z6MkhcbEpJpEvNVDd3n5RurquVdqs5dPU16JDU5VZTDtFgnn",
        "iss": "did:key:z6MkmE8f1sWQoxvMuFC1K3LABAHK8xzdLjGys5ad8AxSNt9i"
    }
}

[olizilla]

Could we toString CIDs before sending them to the stream?

[olizilla]

What is the plan for events like the ObjectCreated from s3 when the CAR is written? it feels like we can only know the state of the system by following the event bus and the ucan log.

My feeling is that we live in a world where not every event has an associated ucan (yet) and we should design for that rather than setting ourselves the task of turning every non ucan event into an invocation. Perhaps we could see this stream as the events stream which includes ucans and non-ucan events.

but... if we wanted to, in the awkward case of store/add we could sign the store/add ucan into the object metadata for the presigned put url, which would then require the user to provide that ucan again as a amz metadata header when making the request... or it could be a new store/put ucan... but whatever it is we'd need it sent in the store/add invocation, so we can get it signed into the url... as we can't otherwise validate it on PUT... having something like that in the object metadata we could then tie the put event back to the user invocation.

[alanshaw]

What is the plan for events like the ObjectCreated from s3 when the CAR is written? it feels like we can only know the state of the system by following the event bus and the ucan log.

My feeling is that we live in a world where not every event has an associated ucan (yet) and we should design for that rather than setting ourselves the task of turning every non ucan event into an invocation. Perhaps we could see this stream as the events stream which includes ucans and non-ucan events.

but... if we wanted to, in the awkward case of store/add we could sign the store/add ucan into the object metadata for the presigned put url, which would then require the user to provide that ucan again as a amz metadata header when making the request... or it could be a new store/put ucan... but whatever it is we'd need it sent in the store/add invocation, so we can get it signed into the url... as we can't otherwise validate it on PUT... having something like that in the object metadata we could then tie the put event back to the user invocation.

I think we should use receipts for this. We already know that we want "events" to be sent back to the user for when certain async operations have happened within the system and I think receipts might be what that is. A series of signed "receipts" allows us to get the accountability we need, but also communicate information back to the user about their upload over a websocket or something.

IMHO this time round we should build the system where things don't really happen without some UCAN tracability, and I'd like to at least see how far we can get with that until it's a big enough pain or simply not possible. In this case I think there is some real utility in generating a UCAN receipt for the put event that can be run through the UCAN log as well as communicated back to the user.

We should definitely add the store/add invocation CID to the meta so we can backfill receipts if needs be.

[vasco-santos]

@olizilla / @alanshaw all the comments above are really related to #83 ... This PR is built on top of it as mentioned above. Let's please keep the context there. I will address comments made here there.

[seed-deploy]

Stack outputs updated

• pr86-w3infra-UploadApiStack

  Name	Value
  ApiEndpoint	https://nx8glk94p5.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr86.up.web3.storage

• pr86-w3infra-BusStack

• pr86-w3infra-CarparkStack

• pr86-w3infra-SatnavStack

• pr86-w3infra-UcanInvocationStack

• pr86-w3infra-UploadDbStack

[vasco-santos]

Looked into npm modules available, but could not get something useful for it. If there are suggestions of modules you know that could help please let me know.

This small function goes through the Object through all its depth and replaces all Link instances by a JSON based on proposal storacha-network/ucanto#171 + multiformats/js-multiformats#228

Once this lands in multiformats, we will be able to drop this adapter function.

[vasco-santos]

@olizilla @alanshaw we should be done here. I think decision we still need to make is whether we want proofs added also in the stream.

Note: CI is currently failing because it cannot install Node.js https://nodejs.org/dist/v16.17.1/node-v16.17.1-linux-x64.tar.xz ...

[seed-deploy]

Stack outputs updated

• pr86-w3infra-UploadApiStack

  Name	Value
  ApiEndpoint	https://gclw8bv170.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr86.up.web3.storage

• pr86-w3infra-BusStack

• pr86-w3infra-CarparkStack

• pr86-w3infra-SatnavStack

• pr86-w3infra-UcanInvocationStack

• pr86-w3infra-UploadDbStack

[vasco-santos]

Added ts property and proof array as discussed with @alanshaw

{
    "carCid": "bafyreibfxctwjqwm5e6mlgr3wgvbazk2srjonlz5a3sj4lyuddxz2qnfau",
    "value": {
        "att": [
            {
                "nb": {
                    "root": {
                        "/": "bafkreibsuf3ruarx3di42cwa2m5h6wxwhoxc5wc46evivzuagabvvwp4za"
                    },
                    "shards": [
                        {
                            "/": "bagbaiera5txbleogmuwm3qrqarhbhomddigohbcsg4ijddvieafn2qcfa43a"
                        }
                    ]
                },
                "can": "upload/add",
                "with": "did:key:z6Mkf4Tc5v4LWessC2cExyQNjprsxJPGfv82pvNmzUXchdPM"
            }
        ],
        "aud": "did:key:z6MkhcbEpJpEvNVDd3n5RurquVdqs5dPU16JDU5VZTDtFgnn",
        "iss": "did:key:z6MkmE8f1sWQoxvMuFC1K3LABAHK8xzdLjGys5ad8AxSNt9i",
        "prf": [
            {
                "/": "bafyreihbmenadjeqpvosrehclwyerqm3sbrmyaqzoj3rzthdkpkvoan6va"
            },
            {
                "/": "bafyreihbmenadjeqpvosrehclwyerqm3sbrmyaqzoj3rzthdkpkvoan6va"
            }
        ]
    },
    "ts": 1670955551515
}

https://us-east-2.console.aws.amazon.com/kinesis/home?region=us-east-2#/streams/details/pr86-w3infra-ucan-stream/dataViewer (latest 2 messages)

[alanshaw]

LGTM