diff --git a/docs/content/docs/getting-started.md b/docs/content/docs/getting-started.md index 2301ced65..337e20997 100644 --- a/docs/content/docs/getting-started.md +++ b/docs/content/docs/getting-started.md @@ -2,7 +2,7 @@ title = "Getting started" description = "Watermill up and running" weight = -9999 -draft = false +draft = false toc = true bref = "Watermill up and running" type = "docs" @@ -10,27 +10,27 @@ type = "docs" ### What is Watermill? -Watermill is a Golang library for working efficiently with message streams. It is intended for building event-driven -applications. It can be used for event sourcing, RPC over messages, sagas, and whatever else comes to your mind. -You can use conventional pub/sub implementations like Kafka or RabbitMQ, but also HTTP or MySQL binlog, if that fits your use case. - -It comes with a set of Pub/Sub implementations and can be easily extended by your own. +Watermill is a Go library for working with message streams. +You can use it to build event-driven systems with popular Pub/Sub implementations like Kafka or RabbitMQ, as well as HTTP or Postgres if that fits your use case. +It comes with a set of Pub/Sub implementations and can be easily extended. Watermill also ships with standard middlewares like instrumentation, poison queue, throttling, correlation, and other tools used by every message-driven application. ### Why use Watermill? -With more projects adopting the microservices pattern over recent years, we realized that synchronous communication -is not always the right choice. Asynchronous methods started to grow as a new standard way to communicate. +When using microservices, synchronous communication is not always the right choice. +Asynchronous methods became a new standard way to communicate. + +While there are many tools and libraries for synchronous communication, like HTTP, correctly setting up +a message-oriented project can be challenging. There are many different message queues and streaming systems, +each with different features, client libraries, and APIs. -But while there's a lot of existing tooling for synchronous integration patterns (e.g. HTTP), correctly setting up -a message-oriented project can be a challenge. There's a lot of different message queues and streaming systems, -each with different features and client library API. +Watermill aims to be the standard messaging library for Go, hiding all that complexity behind an API that is easy to understand. +It provides all you need to build an application based on events or other asynchronous patterns. -Watermill aims to be the standard messaging library for Go, hiding all that complexity behind an API that is easy to -understand. It provides all you might need for building an application based on events or other asynchronous patterns. -After looking at the examples, you should be able to quickly integrate Watermill with your project. +**Watermill is NOT a framework**. +It's a lightweight library that's easy to plug in or remove from your project. ### Install @@ -38,15 +38,27 @@ After looking at the examples, you should be able to quickly integrate Watermill go get -u github.com/ThreeDotsLabs/watermill ``` -### One Minute Background +### One-Minute Background -The basic idea behind event-driven applications stays always the same: listen for incoming messages and react to them. +The idea behind event-driven applications is always the same: listen to and react to incoming messages. Watermill supports this behavior for multiple [publishers and subscribers]({{< ref "/pubsubs" >}}). -The core part of Watermill is the [*Message*]({{< ref "/docs/message" >}}). It is as important as `http.Request` -is for the `http` package. Most Watermill features use this struct in some way. +The core part of Watermill is the [*Message*]({{< ref "/docs/message" >}}). +It is what `http.Request` is for the `net/http` package. +Most Watermill features work with this struct. + +Watermill provides a few APIs for working with messages. +They build on top of each other, each step providing a higher-level API: + +* At the bottom, the `Publisher` and `Subscriber` interfaces. It's the "raw" way of working with messages. You get full control, but also need to handle everything yourself. +* The `Router` is similar to HTTP routers you probably know. It introduces message handlers. +* The `CQRS` component adds generic handlers without needing to marshal and unmarshal messages yourself. + +Watermill components pyramid -Even though PubSub libraries come with complex features, for Watermill it's enough to implement two interfaces to start +## Publisher & Subscriber + +Most Pub/Sub libraries come with complex features. For Watermill, it's enough to implement two interfaces to start working with them: the `Publisher` and `Subscriber`. ```go @@ -63,8 +75,9 @@ type Subscriber interface { ### Subscribing for Messages -Let's start with subscribing. `Subscribe` expects a topic name and returns a channel of incoming messages. -What _topic_ exactly means depends on the PubSub implementation. +`Subscribe` expects a topic name and returns a channel of incoming messages. +What _topic_ exactly means depends on the Pub/Sub implementation. +Usually, it needs to match the topic name used by the publisher. ```go messages, err := subscriber.Subscribe(ctx, "example.topic") @@ -95,15 +108,15 @@ See detailed examples below for supported PubSubs. Running in Docker {{% /collapse-toggle %}} {{% collapse-box id="docker" %}} -The easiest way to run Watermill locally with Kafka is using Docker. +The easiest way to run Watermill locally with Kafka is by using Docker. {{% load-snippet file="src-link/_examples/pubsubs/kafka/docker-compose.yml" type="yaml" %}} The source should go to `main.go`. -To run, execute `docker-compose up` command. +To run, execute the `docker-compose up` command. -A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/). +A more detailed explanation of how it works (and how to add live code reload) can be found in the [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/). {{% /collapse-box %}} {{< /collapse >}} @@ -126,7 +139,7 @@ The easiest way to run Watermill locally with NATS is using Docker. The source should go to `main.go`. -To run execute `docker-compose up` command. +To run, execute the `docker-compose up` command. A more detailed explanation of how it is working (and how to add live code reload) can be found in [*Go Docker dev environment* article](https://threedots.tech/post/go-docker-dev-environment-with-go-modules-and-live-code-reloading/). {{% /collapse-box %}} @@ -145,7 +158,7 @@ A more detailed explanation of how it is working (and how to add live code reloa Running in Docker {{% /collapse-toggle %}} {{% collapse-box id="gcloud-streaming-docker" %}} -You can run Google Cloud Pub/Sub emulator locally for development. +You can run the Google Cloud Pub/Sub emulator locally for development. {{% load-snippet file="src-link/_examples/pubsubs/googlecloud/docker-compose.yml" type="yaml" %}} @@ -209,10 +222,10 @@ A more detailed explanation of how it is working (and how to add live code reloa ### Creating Messages -Watermill doesn't enforce any message format. `NewMessage` expects a slice of bytes as the payload. You can use -strings, JSON, protobuf, Avro, gob, or anything else that serializes to `[]byte`. +Watermill doesn't enforce any message format. `NewMessage` expects a slice of bytes as the payload. +You can use strings, JSON, protobuf, Avro, gob, or anything else that serializes to `[]byte`. -The message UUID is optional, but recommended, as it helps with debugging. +The message UUID is optional but recommended for debugging. ```go msg := message.NewMessage(watermill.NewUUID(), []byte("Hello, world!")) @@ -232,121 +245,109 @@ if err != nil { {{< tabs id="publishing" tabs="go-channel,kafka,nats-streaming,gcloud,amqp,sql" labels="Go Channel,Kafka,NATS Streaming,Google Cloud Pub/Sub,RabbitMQ (AMQP),SQL" >}} {{% tabs-tab id="go-channel"%}} -{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/pubsubs/go-channel/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}} {{% /tabs-tab %}} {{% tabs-tab id="kafka" %}} -{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}} {{% /tabs-tab %}} {{% tabs-tab id="nats-streaming" %}} -{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-streaming/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/pubsubs/nats-streaming/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}} {{% /tabs-tab %}} {{% tabs-tab id="gcloud" %}} -{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/pubsubs/googlecloud/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}} {{% /tabs-tab %}} {{% tabs-tab id="amqp" %}} -{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/pubsubs/amqp/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}} {{% /tabs-tab %}} {{% tabs-tab id="sql" %}} -{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="go process(messages)" last_line_contains="publisher.Publish" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/pubsubs/sql/main.go" first_line_contains="message.NewMessage" last_line_contains="publisher.Publish" padding_after="2" %}} {{% /tabs-tab %}} {{< /tabs >}} -### Using *Message Router* +### Router -[*Publishers and subscribers*]({{< ref "/docs/pub-sub" >}}) are rather low-level parts of Watermill. -In most cases, you'd usually want to use a high-level interface and features like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}). +[*Publishers and subscribers*]({{< ref "/docs/pub-sub" >}}) are the low-level parts of Watermill. +For most cases, you want to use a high-level API: [*Router*]({{< ref "/docs/messages-router" >}}) component. -You might want to send an Ack only if the message was processed successfully. -In other cases, you'll Ack immediately and then worry about processing. -Sometimes, you want to perform some action based on the incoming message, and publish another message in response. +#### Router configuration -To handle these requirements, there is a component named [*Router*]({{< ref "/docs/messages-router" >}}). +Start with configuring the router and adding plugins and middlewares. -### Example application of *Message Router* -The flow of the example application looks like this: +A middleware is a function executed for each incoming message. +You can use one of the existing ones for things like [correlation, metrics, poison queue, retrying, throttling, etc.]({{< ref "/docs/messages-router#middleware" >}}). +You can also create your own. + +{{% render-md %}} +{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="message.NewRouter" last_line_contains="middleware.Recoverer," padding_after="1" %}} +{{% /render-md %}} -1. A message is produced on topic `incoming_messages_topic` every second. -2. `struct_handler` handler listens on `incoming_messages_topic`. When a message is received, the UUID is printed and a new message is produced on `outgoing_messages_topic`. -3. `print_incoming_messages` handler listens on `incoming_messages_topic` and prints the messages' UUID, payload and metadata. -4. `print_outgoing_messages` handler listens on `outgoing_messages_topic` and prints the messages' UUID, payload and metadata. Correlation ID should be the same as in the message on `incoming_messages_topic`. +#### Handlers -#### Router configuration +Set up handlers that the router uses. +Each handler independently handles incoming messages. -Start with configuring the router, adding plugins and middlewares. -Then set up handlers that the router will use. Each handler will independently handle messages. +A handler listens to messages from the given subscriber and topic. +Any messages returned from the handler function will be published to the given publisher and topic. {{% render-md %}} -{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="package" last_line_contains="router.Run(ctx)" padding_after="4" %}} +{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="AddHandler returns" last_line_contains=")" padding_after="0" %}} {{% /render-md %}} -#### Incoming messages +*Note: the example above uses one `pubSub` argument for both the subscriber and publisher. +It's because we use the `GoChannel` implementation, which is a simple in-memory Pub/Sub.* -The `struct_handler` consumes messages from `incoming_messages_topic`, so we are simulating incoming traffic by calling `publishMessages()` in the background. -Notice that we've added the `SetCorrelationID` middleware. A Correlation ID will be added to all messages produced by the router (it will be stored in metadata). +Alternatively, if you don't plan to publish messages from within the handler, you can use the simpler `AddNoPublisherHandler` method. {{% render-md %}} -{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="func publishMessages" last_line_contains="time.Sleep(time.Second)" padding_after="2" %}} +{{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="AddNoPublisherHandler" last_line_contains=")" padding_after="0" %}} {{% /render-md %}} -#### Handlers - -You may have noticed that there are two types of *handler functions*: +You can use two types of *handler functions*: -1. function `func(msg *message.Message) ([]*message.Message, error)` -2. method `func (c structHandler) Handler(msg *message.Message) ([]*message.Message, error)` +1. a function `func(msg *message.Message) ([]*message.Message, error)` +2. a struct method `func (c structHandler) Handler(msg *message.Message) ([]*message.Message, error)` -If your handler is a function without any dependencies, it's fine to use the first one. -The second option is useful when your handler requires some dependencies like database handle, a logger, etc. +Use the first one if your handler is a function without any dependencies. +The second option is useful when your handler requires dependencies such as a database handle or a logger. {{% render-md %}} {{% load-snippet-partial file="src-link/_examples/basic/3-router/main.go" first_line_contains="func printMessages" last_line_contains="return message.Messages{msg}, nil" padding_after="3" %}} {{% /render-md %}} -#### Done! - -You can run this example by `go run main.go`. - -You've just created your first application with Watermill. You can find the full source in [/_examples/basic/3-router/main.go](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/3-router/main.go). +The complete example's source can be found at [/_examples/basic/3-router/main.go](https://github.com/ThreeDotsLabs/watermill/blob/master/_examples/basic/3-router/main.go). ### Logging -To see Watermill's logs, you have to pass any logger that implements the [LoggerAdapter](https://github.com/ThreeDotsLabs/watermill/blob/master/log.go). +To see Watermill's logs, pass any logger that implements the [LoggerAdapter](https://github.com/ThreeDotsLabs/watermill/blob/master/log.go). For experimental development, you can use `NewStdLogger`. -### Testing - -Watermill provides [a set of test scenarios](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go) -that any Pub/Sub implementation can use. Each test suite needs to declare what features it supports and how to construct a new Pub/Sub. -These scenarios check both basic usage and more uncommon use cases. Stress tests are also included. - -### Deployment - -Watermill is not a framework. We don't enforce any type of deployment and it's totally up to you. +## What's next? -### What's next? +For more details, see [documentation topics]({{< ref "/docs" >}}). -For more detailed documentation check [documentation topics]({{< ref "/docs" >}}). +See the [CQRS component](/docs/cqrs) for another high-level API. -#### Examples +## Examples Check out the [examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples) that will show you how to start using Watermill. The recommended entry point is [Your first Watermill application](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/1-your-first-app). -It contains the entire environment in `docker-compose.yml`, including Golang and Kafka, which you can run with one command. +It contains the entire environment in `docker-compose.yml`, including Go and Kafka, which you can run with one command. After that, you can see the [Realtime feed](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/basic/2-realtime-feed) example. -It uses more middlewares and contains two handlers. There is also a separate application for publishing messages. +It uses more middlewares and contains two handlers. -For a different subscriber implementation, namely **HTTP**, refer to the [receiving-webhooks](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/receiving-webhooks) example. It is a very simple application that saves webhooks to Kafka. +For a different subscriber implementation (**HTTP**), see the [receiving-webhooks](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/real-world-examples/receiving-webhooks) example. +It is a straightforward application that saves webhooks to Kafka. -Full list of examples can be found in the project's [README](https://github.com/ThreeDotsLabs/watermill#examples). +You can find the complete list of examples in the [README](https://github.com/ThreeDotsLabs/watermill#examples). -#### Support +## Support -If anything is not clear, feel free to use any of our [support channels]({{< ref "/support" >}}), we will be glad to help. +If anything is not clear, feel free to use any of our [support channels]({{< ref "/support" >}}); we will be glad to help. diff --git a/docs/content/docs/pub-sub-implementing.md b/docs/content/docs/pub-sub-implementing.md index d0cfe03c6..6531014ae 100644 --- a/docs/content/docs/pub-sub-implementing.md +++ b/docs/content/docs/pub-sub-implementing.md @@ -16,6 +16,12 @@ To add support for a custom Pub/Sub, you have to implement both `message.Publish {{% load-snippet-partial file="src-link/message/pubsub.go" first_line_contains="type Publisher interface" last_line_contains="type SubscribeInitializer" padding_after="0" %}} {{% /render-md %}} +### Testing + +Watermill provides [a set of test scenarios](https://github.com/ThreeDotsLabs/watermill/blob/master/pubsub/tests/test_pubsub.go) +that any Pub/Sub implementation can use. Each test suite needs to declare what features it supports and how to construct a new Pub/Sub. +These scenarios check both basic usage and more uncommon use cases. Stress tests are also included. + ### TODO list Here are a few things you shouldn't forget about: diff --git a/docs/layouts/partials/footer.html b/docs/layouts/partials/footer.html index a07b3a0d9..27a398d7b 100644 --- a/docs/layouts/partials/footer.html +++ b/docs/layouts/partials/footer.html @@ -20,7 +20,9 @@
-

Help us improve this page

+ {{ with .Page.File }} +

Help us improve this page

+ {{ end }}

©Three Dots Labs MIT License

diff --git a/docs/static/img/pyramid.png b/docs/static/img/pyramid.png new file mode 100644 index 000000000..3507fc74a Binary files /dev/null and b/docs/static/img/pyramid.png differ