-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add kinesis log stream #86
Conversation
View stack outputs
|
c23e572
to
1b7fd7a
Compare
1b7fd7a
to
dad7f38
Compare
dad7f38
to
3e2f4b2
Compare
3e2f4b2
to
fba9f63
Compare
fba9f63
to
9618aef
Compare
EDIT: SEE COMMENTS LATER AS THIS WAS MODIFIED Data written into kinesis stream {
"carCid": "bafyreibr5w4fjaxg5da5gwovujsz6vmxp4dwtsm73iiiwki7ym5gon7p5q",
"data": {
"att": [
{
"nb": {
"link": {
"code": 514,
"version": 1,
"hash": {
"0": 18,
"1": 32,
"2": 132,
"3": 245,
"4": 3,
"5": 183,
"6": 72,
"7": 151,
"8": 231,
"9": 183,
"10": 40,
"11": 33,
"12": 244,
"13": 229,
"14": 22,
"15": 202,
"16": 14,
"17": 50,
"18": 153,
"19": 111,
"20": 251,
"21": 187,
"22": 157,
"23": 131,
"24": 194,
"25": 123,
"26": 227,
"27": 195,
"28": 143,
"29": 11,
"30": 74,
"31": 2,
"32": 43,
"33": 98
}
},
"size": 225
},
"can": "store/add",
"with": "did:key:z6Mkf4Tc5v4LWessC2cExyQNjprsxJPGfv82pvNmzUXchdPM"
}
],
"aud": "did:key:z6MkhcbEpJpEvNVDd3n5RurquVdqs5dPU16JDU5VZTDtFgnn",
"iss": "did:key:z6MkmE8f1sWQoxvMuFC1K3LABAHK8xzdLjGys5ad8AxSNt9i"
}
} |
9618aef
to
3a0c00b
Compare
Could we toString CIDs before sending them to the stream? |
What is the plan for events like the ObjectCreated from s3 when the CAR is written? it feels like we can only know the state of the system by following the event bus and the ucan log. My feeling is that we live in a world where not every event has an associated ucan (yet) and we should design for that rather than setting ourselves the task of turning every non ucan event into an invocation. Perhaps we could see this stream as the events stream which includes ucans and non-ucan events. but... if we wanted to, in the awkward case of |
I think we should use receipts for this. We already know that we want "events" to be sent back to the user for when certain async operations have happened within the system and I think receipts might be what that is. A series of signed "receipts" allows us to get the accountability we need, but also communicate information back to the user about their upload over a websocket or something. IMHO this time round we should build the system where things don't really happen without some UCAN tracability, and I'd like to at least see how far we can get with that until it's a big enough pain or simply not possible. In this case I think there is some real utility in generating a UCAN receipt for the put event that can be run through the UCAN log as well as communicated back to the user. We should definitely add the store/add invocation CID to the meta so we can backfill receipts if needs be. |
3a0c00b
to
8a8af5c
Compare
8a8af5c
to
f0655d6
Compare
Stack outputs updated
|
/** | ||
* @param {any} value | ||
*/ | ||
export const replaceAllLinkValues = (value) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looked into npm modules available, but could not get something useful for it. If there are suggestions of modules you know that could help please let me know.
This small function goes through the Object through all its depth and replaces all Link
instances by a JSON based on proposal storacha-network/ucanto#171 + multiformats/js-multiformats#228
Once this lands in multiformats
, we will be able to drop this adapter function.
7044083
to
3f5346e
Compare
df23ab9
to
bebfd29
Compare
bebfd29
to
51bf234
Compare
@olizilla @alanshaw we should be done here. I think decision we still need to make is whether we want proofs added also in the stream.
|
Stack outputs updated
|
c22273c
to
31a4aed
Compare
Added {
"carCid": "bafyreibfxctwjqwm5e6mlgr3wgvbazk2srjonlz5a3sj4lyuddxz2qnfau",
"value": {
"att": [
{
"nb": {
"root": {
"/": "bafkreibsuf3ruarx3di42cwa2m5h6wxwhoxc5wc46evivzuagabvvwp4za"
},
"shards": [
{
"/": "bagbaiera5txbleogmuwm3qrqarhbhomddigohbcsg4ijddvieafn2qcfa43a"
}
]
},
"can": "upload/add",
"with": "did:key:z6Mkf4Tc5v4LWessC2cExyQNjprsxJPGfv82pvNmzUXchdPM"
}
],
"aud": "did:key:z6MkhcbEpJpEvNVDd3n5RurquVdqs5dPU16JDU5VZTDtFgnn",
"iss": "did:key:z6MkmE8f1sWQoxvMuFC1K3LABAHK8xzdLjGys5ad8AxSNt9i",
"prf": [
{
"/": "bafyreihbmenadjeqpvosrehclwyerqm3sbrmyaqzoj3rzthdkpkvoan6va"
},
{
"/": "bafyreihbmenadjeqpvosrehclwyerqm3sbrmyaqzoj3rzthdkpkvoan6va"
}
]
},
"ts": 1670955551515
} https://us-east-2.console.aws.amazon.com/kinesis/home?region=us-east-2#/streams/details/pr86-w3infra-ucan-stream/dataViewer (latest 2 messages) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Co-authored-by: Alan Shaw <[email protected]>
Adds kinesis ucan log stream to ucanto service. Once a UCAN invocation is handled by the service, it is sent to Amazon Kinesis data streams for post processing (JSON with invocation CID, invocation bytes, and decoded invocation).
Kinesis log stream has its own stack named
UcanStreamStack
which will include resources needed for post processing of ucan stream ops.ApiStack
depends onUcanStreamStack
given it will use its stream, as well as its data further down the line to get content like user facing statsPer https://www.notion.so/UCAN-LOG-0f3870fc4b404f5cbf646bf16b463365
Implementation details:
{ carCid: string, value: { att: UCAN.Capabilities, aud: 'did:${string}:${string}', iss: 'did:${string}:${string}' } }
prf
,exp
,nbf
,fct
,nnc
,v
, andsignature
.Other notes:
Follow ups:
Needs: