WIP Bridge: js-module sender input #1004
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
svix-onelson
force-pushed
the
onelson/bridge-poller
branch
from
4e94dda to
a986265
Compare
The general idea here is to let folks define a js module to use as a data source for a "sender." This could be general purpose but the initial use case is for polling APIs that don't expose webhooks. Currently this diff contains only support code for config handling. Some implementation details were worked out separately as a [POC in a separate repo.](https://github.com/svix-onelson/poller-input-poc/) By using `fetch` to issue HTTP requests, we can track state in the module itself to decide if the data is worth sending a webhook for. Example: <https://github.com/svix-onelson/poller-input-poc/blob/main/fetcher.js> There are many barriers to integrating the code in the POC linked above which need to be cleared first. We need: - a custom module loader to let us source modules from the bridge config instead of js files on disk. - Glue code to connect the `op_forward` deno extension calls to either a transformation or svix sender output. - existing transformation code in bridge needs to be refactored to allow us to use "newer deno" without also introducing a memory leak. For the latter, <svix/monorepo-private#5670> aims to solve this. In addition to the above, deno ops (i.e native extension code) are supposed to be able to register state with the runtime, but I wasn't able to get it to work for keeping track of which worker was which (allowing us to propagate payloads to the appropriate output).
svix-onelson
force-pushed
the
onelson/bridge-poller
branch
from
a986265 to
f926947
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The general idea here is to let folks define a js module to use as a
data source for a "sender."
This could be general purpose but the initial use case is for polling
APIs that don't expose webhooks.
Currently, this diff contains only support code for config handling.
Some implementation details were worked out separately as a
POC in a separate repo.
By using
fetchto issue HTTP requests, we can track state in themodule itself to decide if the data is worth sending a webhook for.
Example:
https://github.com/svix-onelson/poller-input-poc/blob/main/fetcher.js
There are many barriers to integrating the code in the POC linked above
which need to be cleared first.
We need:
bridge config instead of js files on disk.
op_forwarddeno extension calls to either atransformation or svix sender output.
us to use "newer deno" without also introducing a memory leak.
For the latter, https://github.com/svix/monorepo-private/issues/5670
aims to solve this.
In addition to the above, deno ops (i.e native extension code) are
supposed to be able to register state with the runtime, but I wasn't
able to get it to work for keeping track of which worker was which
(allowing us to propagate payloads to the appropriate output).