feat: kinesis delivery stream updates #237
Conversation
and add more fields to the out and value structs to enable metrics prototyping
thanks to the "forbidden characters" section of https://aws.amazon.com/blogs/big-data/create-tables-in-amazon-athena-from-nested-json-and-mappings-using-jsonserde/ also remove the paths parameter because it does nothing!! https://stackoverflow.com/questions/42170339/what-does-with-serdeproperties-paths-key1-key2-key3-really-do-in-h?noredirect=1&lq=1
View stack outputs
|
|
aha yea, tbh this is merging into @vasco-santos's PR and I mostly put it up for him to peek at before I pushed on top of #191 but this feedback is super helpful - I need to add some comments for the more esoteric stuff! |
based on the queries I've put together this week
this seems to behave better with deleting stacks
| @@ -145,8 +147,11 @@ export function UcanFirehoseStack({ stack, app }) { | |||
| }, | |||
| ], | |||
| }, | |||
| // See https://docs.aws.amazon.com/athena/latest/ug/partition-projection-kinesis-firehose-example.html | |||
| prefix: 'logs/issuer=!{partitionKeyFromQuery:issuer}/', | |||
There was a problem hiding this comment.
I think my logic here was to enable performant queries per issuer.
However, after reading through requirements docs, I think we should partition as you suggest instead. We can still query by issuer, but within a range, instead of all time. Which is good, given we already save space metrics anyway :)
stacks/firehose-stack.js
Outdated
| parameters: { | ||
| classification: "json", | ||
| typeOfData: "file", | ||
| // See See https://docs.aws.amazon.com/athena/latest/ug/partition-projection-kinesis-firehose-example.html for more information on projection |
There was a problem hiding this comment.
| // See See https://docs.aws.amazon.com/athena/latest/ug/partition-projection-kinesis-firehose-example.html for more information on projection | |
| // @see https://docs.aws.amazon.com/athena/latest/ug/partition-projection-kinesis-firehose-example.html for more information on projection |
travis
force-pushed
the
feat/kinesis-delivery-stream-updates
branch
from
37a476a
to
6b8504a
Compare
merging into #191 - mostly put this up so @vasco-santos can review before I push it into the PR he opened!
valueandoutstruct definitions to allow for more interesting queries