Bypass kinesis #111
Open
Bypass kinesis #111
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
December 8, 2020 12:55
…s. This is in response to the Nov 2020 AWS Kinesis outage. Usage: leo.write and leo.load now accept `option.disable_kinesis: true` or you can use the environment variable `DISABLE_KINESIS=true` Note: Bypassing kinises can create a race condition if 2 bots are writing to the same queue. The race happens in the bot reading from that queue. If the bot reads too quickly it will checkpoint and potentially skip events that were created by the seconds source bot but still being written to the DB. A workaround is to not read all the way to the current timestamp as this will allow the in flight events to land in the correct place before being read. Also, when there are 2 source bots there is the possibility that they generate the same eid. This would happen when both bots begin the kinesis stream processor code on the same millisecond. There is no workaround for this currently. However, ideas may include giving each writing bot a block record range for the eid, extending the eid to include an partition/writer identifier between the timestamp and record number, making queue partitions a first class feature This does NOT solve the case of firehose. firehose is used to collect the events over the course of 1 mintue and then it emits an event to kinesis. A workaround for firehose is to stop sending to firehose and bypass kinesis.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allows for the ability to bypass kinesis and write directly to the bus. This is in response to the Nov 2020 AWS Kinesis outage.
Usage:
leo.write and leo.load now accept
option.disable_kinesis: trueor you can use the environment variableDISABLE_KINESIS=trueNote:
Bypassing kinises can create a race condition if 2 bots are writing to the same queue. The race happens in the bot reading from that queue. If the bot reads too quickly it will checkpoint and potentially skip events that were created by the seconds source bot but still being written to the DB.
A workaround is to not read all the way to the current timestamp as this will allow the in flight events to land in the correct place before being read.
Also, when there are 2 source bots there is the possibility that they generate the same eid. This would happen when both bots begin the kinesis stream processor code on the same millisecond. There is no workaround for this currently. However, ideas may include giving each writing bot a block record range for the eid, extending the eid to include an partition/writer identifier between the timestamp and record number, making queue partitions a first class feature
This does NOT solve the case of firehose. firehose is used to collect the events over the course of 1 mintue and then it emits an event to kinesis. A workaround for firehose is to stop sending to firehose and bypass kinesis.