Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNM: kinesis support for bridge #924

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
renaming svix-agent to svix-webhook-bridge
svix-onelson committed May 8, 2023
commit cbe04b74717f7ca339537034acb6257602433440
42 changes: 21 additions & 21 deletions webhook-bridge/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions webhook-bridge/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[workspace]
members = [
"generic-queue",
"svix-agent-types",
"svix-agent",
"svix-agent-plugin-generic",
"svix-agent-plugin-webhook-receiver",
"svix-webhook-bridge-types",
"svix-webhook-bridge",
"svix-webhook-bridge-plugin-queue-consumer",
"svix-webhook-bridge-plugin-webhook-receiver",
]
39 changes: 20 additions & 19 deletions webhook-bridge/Dockerfile
Original file line number Diff line number Diff line change
@@ -23,28 +23,28 @@ WORKDIR /app
COPY Cargo.toml .
COPY Cargo.lock .
COPY generic-queue/Cargo.toml generic-queue/
COPY svix-agent-types/Cargo.toml svix-agent-types/
COPY svix-agent-plugin-generic/Cargo.toml svix-agent-plugin-generic/
COPY svix-agent-plugin-webhook-receiver/Cargo.toml svix-agent-plugin-webhook-receiver/
COPY svix-agent/Cargo.toml svix-agent/
COPY svix-webhook-bridge-types/Cargo.toml svix-webhook-bridge-types/
COPY svix-webhook-bridge-plugin-queue-consumer/Cargo.toml svix-webhook-bridge-plugin-queue-consumer/
COPY svix-webhook-bridge-plugin-webhook-receiver/Cargo.toml svix-webhook-bridge-plugin-webhook-receiver/
COPY svix-webhook-bridge/Cargo.toml svix-webhook-bridge/
RUN set -ex ;\
mkdir generic-queue/src ;\
mkdir svix-agent-plugin-generic/src ;\
mkdir svix-agent-plugin-webhook-receiver/src ;\
mkdir svix-agent-types/src ;\
mkdir svix-agent/src ;\
mkdir svix-webhook-bridge-plugin-queue-consumer/src ;\
mkdir svix-webhook-bridge-plugin-webhook-receiver/src ;\
mkdir svix-webhook-bridge-types/src ;\
mkdir svix-webhook-bridge/src ;\
echo '' > generic-queue/src/lib.rs ;\
echo '' > svix-agent-plugin-generic/src/lib.rs ;\
echo '' > svix-agent-plugin-webhook-receiver/src/lib.rs ;\
echo '' > svix-agent-types/src/lib.rs ;\
echo 'fn main() { println!("Dummy!"); }' > svix-agent/src/main.rs ;\
echo '' > svix-webhook-bridge-plugin-queue-consumer/src/lib.rs ;\
echo '' > svix-webhook-bridge-plugin-webhook-receiver/src/lib.rs ;\
echo '' > svix-webhook-bridge-types/src/lib.rs ;\
echo 'fn main() { println!("Dummy!"); }' > svix-webhook-bridge/src/main.rs ;\
cargo build --release ;\
rm -rf \
generic-queue/src \
svix-agent-plugin-generic/src \
svix-agent-plugin-webhook-receiver/src \
svix-agent-types/src \
svix-agent/src
svix-webhook-bridge-plugin-queue-consumer/src \
svix-webhook-bridge-plugin-webhook-receiver/src \
svix-webhook-bridge-types/src \
svix-webhook-bridge/src

COPY . .
# touching the lib.rs/main.rs ensures cargo rebuilds them instead of considering them already built.
@@ -68,7 +68,8 @@ RUN apt-get update ;\

USER appuser

COPY --from=build /app/target/release/svix-agent /usr/local/bin/svix-agent
COPY --from=build /app/target/release/svix-webhook-bridge /usr/local/bin/svix-webhook-bridge

# Will fail if there's no `svix-agent.yaml` in the CWD or `SVIX_AGENT_CFG` is not set to a valid path to a config
CMD ["svix-agent"]
# Will fail if there's no `svix-webhook-bridge.yaml` in the CWD or `SVIX_WEBHOOK_BRIDGE_CFG` is not set to a valid
# path to a config.
CMD ["svix-webhook-bridge"]
22 changes: 11 additions & 11 deletions webhook-bridge/README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
# Svix Agent
# Svix Webhook Bridge

This service subscribes to a queue or stream and forwards each item to Svix when a valid message is found.

## Usage

```
svix-agent -c path/to/svix-agent.yaml
svix-webhook-bridge -c path/to/svix-webhook-bridge.yaml
```

## Configuration

> For an annotated sample configuration see [the example config](svix-agent.example.yaml).
> For an annotated sample configuration see [the example config](svix-webhook-bridge.example.yaml).

`svix-agent` is organized in terms of "plugins" which are tasks that run in tandem.
Each plugin represents a unit of work performed while the agent while it runs.
`svix-webhook-bridge` is organized in terms of "plugins" which are tasks that run in tandem.
Each plugin represents a unit of work performed while the service runs.

Presently there are 2 "plugins" available for `svix-agent`.
Presently there are 2 "plugins" available for `svix-webhook-bridge`.

### svix-agent-plugin-generic
### svix-webhook-bridge-plugin-queue-consumer

This plugin consumes messages from message queues to and forwards them to Svix to create messages.

@@ -56,7 +56,7 @@ Messages received by these consumers must follow an expected format:
For detail on the `message` field, see: <https://api.svix.com/docs#tag/Message/operation/v1.message.create>

Important to note that queues, exchanges, topics, or what have you, should be created and configured independently,
prior to using the agent plugin. There's nothing in place to automatically create these resources.
prior to using the plugin. There's nothing in place to automatically create these resources.
The plugin will only try (and fail) to read from the stream in such a case.


@@ -129,7 +129,7 @@ Note that the SQS consumer requires credentials to be set as environment vars:
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`

> This incidentally means all SQS consumers configured for a given `svix-agent` will need to share these details.
> This incidentally means all SQS consumers configured for a given `svix-webhook-bridge` will need to share these details.

```yaml
plugins:
@@ -145,12 +145,12 @@ plugins:
```


### svix-agent-plugin-webhook-receiver
### svix-webhook-bridge-plugin-webhook-receiver

This plugin starts an HTTP server which accepts webhooks and forwards them to one of the supported messaging
systems.

Again, same as with `svix-agent-plugin-generic`, the supported systems are:
Again, same as with `svix-webhook-bridge-plugin-queue-consumer`, the supported systems are:

- GCP Pub/Sub
- RabbitMQ
2 changes: 1 addition & 1 deletion webhook-bridge/generic-queue/src/gcp_pubsub.rs
Original file line number Diff line number Diff line change
@@ -222,7 +222,7 @@ impl<T: 'static + Serialize + Send + Sync> TaskQueueSend<T> for GCPPubSubQueuePr
let topic = self.client.topic(&self.topic);

// Publishing to a non-existent topic will cause the publisher to wait (forever?)
// Giving this error will allow dependents like `svix-agent-plugin-webhook-receiver` to
// Giving this error will allow dependents like `svix-webhook-bridge-plugin-webhook-receiver` to
// respond 500 immediately when this happens, instead of holding the connection open
// indefinitely.
if !topic.exists(None).await.map_err(QueueError::generic)? {
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "svix-agent-plugin-generic"
name = "svix-webhook-bridge-plugin-queue-consumer"
version = "0.1.0"
edition = "2021"

@@ -11,7 +11,7 @@ generic_queue = { path = "../generic-queue" }
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
svix = "0.84.1"
svix-agent-types = { path = "../svix-agent-types" }
svix-webhook-bridge-types = { path = "../svix-webhook-bridge-types" }
tokio = { version = "1", features = ["full"] }
tokio-executor-trait = "2.1"
tokio-reactor-trait = "1.1"
Original file line number Diff line number Diff line change
@@ -9,7 +9,7 @@ use generic_queue::gcp_pubsub::{
use generic_queue::{Delivery, TaskQueueBackend, TaskQueueReceive};
use std::time::{Duration, Instant};
use svix::api::Svix;
use svix_agent_types::{async_trait, Plugin};
use svix_webhook_bridge_types::{async_trait, Plugin};
use tracing::instrument;

pub struct GCPPubSubConsumerPlugin {
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ use generic_queue::{
};
use serde::{Deserialize, Serialize};
use svix::api::{MessageIn, PostOptions as PostOptions_, Svix};
use svix_agent_types::{async_trait, Plugin};
use svix_webhook_bridge_types::{async_trait, Plugin};

pub mod config;
pub use config::{
Original file line number Diff line number Diff line change
@@ -11,12 +11,12 @@ use std::time::Duration;

use serde_json::json;
use svix::api::MessageIn;
use svix_agent_plugin_generic::config::GCPPubSubInputOpts;
use svix_agent_plugin_generic::{
use svix_webhook_bridge_plugin_queue_consumer::config::GCPPubSubInputOpts;
use svix_webhook_bridge_plugin_queue_consumer::{
config::{OutputOpts, SvixOptions},
CreateMessageRequest, GCPPubSubConsumerConfig, GCPPubSubConsumerPlugin,
};
use svix_agent_types::Plugin;
use svix_webhook_bridge_types::Plugin;
use wiremock::matchers::method;
use wiremock::{Mock, MockServer, ResponseTemplate};

Original file line number Diff line number Diff line change
@@ -7,11 +7,11 @@ use lapin::{options::QueueDeclareOptions, Channel, Connection, ConnectionPropert
use serde_json::json;
use std::time::Duration;
use svix::api::MessageIn;
use svix_agent_plugin_generic::{
use svix_webhook_bridge_plugin_queue_consumer::{
config::{OutputOpts, RabbitMqInputOpts, SvixOptions},
CreateMessageRequest, RabbitMqConsumerConfig, RabbitMqConsumerPlugin,
};
use svix_agent_types::Plugin;
use svix_webhook_bridge_types::Plugin;
use wiremock::matchers::method;
use wiremock::{Mock, MockServer, ResponseTemplate};

Original file line number Diff line number Diff line change
@@ -6,11 +6,11 @@ use std::time::Duration;
use redis::{AsyncCommands, Client};
use serde_json::json;
use svix::api::MessageIn;
use svix_agent_plugin_generic::{
use svix_webhook_bridge_plugin_queue_consumer::{
config::{OutputOpts, SvixOptions},
CreateMessageRequest, RedisConsumerConfig, RedisConsumerPlugin, RedisInputOpts,
};
use svix_agent_types::Plugin;
use svix_webhook_bridge_types::Plugin;
use wiremock::matchers::method;
use wiremock::{Mock, MockServer, ResponseTemplate};

Original file line number Diff line number Diff line change
@@ -8,11 +8,11 @@ use std::time::Duration;
use aws_sdk_sqs::Client;
use serde_json::json;
use svix::api::MessageIn;
use svix_agent_plugin_generic::{
use svix_webhook_bridge_plugin_queue_consumer::{
config::{OutputOpts, SvixOptions},
CreateMessageRequest, SqsConsumerConfig, SqsConsumerPlugin, SqsInputOpts,
};
use svix_agent_types::Plugin;
use svix_webhook_bridge_types::Plugin;
use wiremock::matchers::method;
use wiremock::{Mock, MockServer, ResponseTemplate};

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "svix-agent-plugin-webhook-receiver"
name = "svix-webhook-bridge-plugin-webhook-receiver"
version = "0.1.0"
edition = "2021"

@@ -15,7 +15,7 @@ serde = { version = "1", features = ["derive"] }
serde_json = "1"
smol = "1.3"
svix = "0.84.1"
svix-agent-types = { path = "../svix-agent-types" }
svix-webhook-bridge-types = { path = "../svix-webhook-bridge-types" }
threadpool = "1"
tracing = "0.1"
tokio = { version = "1", features = ["full"] }
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ use axum::{
};
use serde::Deserialize;
use std::net::SocketAddr;
use svix_agent_types::{async_trait, Plugin};
use svix_webhook_bridge_types::{async_trait, Plugin};
use tracing::instrument;
use types::{IntegrationId, IntegrationState, InternalState, SerializableRequest, Unvalidated};

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "svix-agent-types"
name = "svix-webhook-bridge-types"
version = "0.1.0"
edition = "2021"

Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@
# The OpenTelemetry address to send events to if given.
#opentelemetry_address: "http://localhost:1234"

# The OpenTelemetry service name to use. Default: "svix-agent"
# The OpenTelemetry service name to use. Default: "svix-webhook-bridge"
# If the OpenTelemetry address is not set, this will do nothing.
#opentelemetry_service_name: "my-agent"

Loading