feat: Kinesis delivery stream and Athena query infrastructure to enab…

…le queries over the UCAN logs (#191)

This PR has an implementation of option 2 from
#190 (comment)

This encompasses a fair amount of functionality - partitioning the UCAN
logs into S3 buckets, configuring a Glue database and tables, adding
example queries to Athena and more. A partial list of functionality
follow:

- implement UCAN log partitioning in S3
- first partition by `type` - everything in "workflows" shows up in
"receipts" so this reduces the amount of data scanned by ~50%
- next partition by `op` to allow us to create tables that only query a
specific operation (ie, `store/add` or `provider/add`) - this lets us
add operation-specific Glue table schemas with much less clutter in
result types than we'd need if we tried to defined all possible inputs
and outputs in a single table
  - finally partition by date to allow queries to only load recent data

- use these partitions to implement standalone tables for receipts in
general and the `store/add`, `upload/add` and `provider/add` UCANs
specifically,
- add queries that demonstrate the use of all of these tables
- add dynamo connector so we can join the UCAN logs to our Dynamo tables
in queries
- add queries that demonstrate using the Dynamo and Glue tables together

---------

Co-authored-by: Travis Vachon <[email protected]>
Co-authored-by: Travis Vachon <[email protected]>

package.json

@@ -16,7 +16,6 @@
     "test-integration": "ava --verbose --timeout=120s test/*.test.js",
     "fetch-metrics-for-space": "npm run fetch-metrics-for-space -w tools",
     "d1-dynamo-migration": "npm run d1-dynamo-migration -w tools"
-
   },
   "devDependencies": {
     "@ipld/car": "^5.1.0",

stacks/config.js

@@ -15,6 +15,18 @@ export function getBucketName (name, stage, version = 0) {
   return `${name}-${stage}-${version}`
 }
 
+/**
+ * Get nicer CDK resources name
+ *
+ * @param {string} name
+ * @param {string} stage
+ * @param {number} version
+ */
+export function getCdkNames (name, stage, version = 0) {
+  // e.g `prod-w3infra-ucan-stream-delivery-0`
+  return `${stage}-w3infra-${name}-${version}`
+}
+
 /**
  * Is an ephemeral build?
  *