Skip to content

Commit

Permalink
fix: reduce costs on kinesis streams (#231)
Browse files Browse the repository at this point in the history
We are currently being billed quite a bit in AWS for all the Kinesis
streams we keep open for dev/CI.

This PR addresses this by:
- not deploying 2 kinesis streams per env to keep historical content. We
really just need the production stream still deployed until we read it.
- we can also decrease the persistence of the streams to keep costs down

As follow ups, we should also consider (I will create issues):
- tool that proactively tears down running infrastructure for stalled
PRs (or setup tests in a way that kinesis is put on and off for the
tests)
- get historical content kinesis stream flushed out
  • Loading branch information
vasco-santos authored Sep 19, 2023
1 parent dcaeac3 commit 7803885
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 11 deletions.
15 changes: 15 additions & 0 deletions stacks/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,21 @@ export function getKinesisEventSourceConfig (stack) {
}
}

/**
* @param {import('@serverless-stack/resources').Stack} stack
*/
export function getKinesisStreamConfig (stack) {
if (stack.stage !== 'production') {
return {
retentionPeriod: Duration.hours(24)
}
}

return {
retentionPeriod: Duration.days(365)
}
}

export function getApiPackageJson () {
// @ts-expect-error ts thinks this is unused becuase of the ignore
const require = createRequire(import.meta.url)
Expand Down
21 changes: 10 additions & 11 deletions stacks/ucan-invocation-stack.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ import {
Queue,
use
} from '@serverless-stack/resources'
import { Duration } from 'aws-cdk-lib'

import { CarparkStack } from './carpark-stack.js'
import { UploadDbStack } from './upload-db-stack.js'
import {
getBucketConfig,
getKinesisEventSourceConfig,
getKinesisStreamConfig,
setupSentry
} from './config.js'

Expand Down Expand Up @@ -190,20 +190,19 @@ export function UcanInvocationStack({ stack, app }) {
})

// TODO: keep for historical content that we might want to process
new KinesisStream(stack, 'ucan-stream', {
cdk: {
stream: {
retentionPeriod: Duration.days(365)
}
},
})
// only needed for production
if (stack.stage === 'production') {
new KinesisStream(stack, 'ucan-stream', {
cdk: {
stream: getKinesisStreamConfig(stack)
},
})
}

// create a kinesis stream
const ucanStream = new KinesisStream(stack, 'ucan-stream-v2', {
cdk: {
stream: {
retentionPeriod: Duration.days(365)
}
stream: getKinesisStreamConfig(stack)
},
consumers: {
metricsStoreAddTotalConsumer: {
Expand Down

0 comments on commit 7803885

Please sign in to comment.