Skip to content

Commit

Permalink
added AWS v1.0 docs (#506)
Browse files Browse the repository at this point in the history
  • Loading branch information
@roblaszczak
roblaszczak authored Oct 28, 2024
1 parent 573eef2 commit 94c0209
Show file tree
Hide file tree
Showing 6 changed files with 256 additions and 11 deletions.
1 change: 1 addition & 0 deletions docs/build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ cloneOrPull "https://github.com/ThreeDotsLabs/watermill-sql.git" content/src-lin
cloneOrPull "https://github.com/ThreeDotsLabs/watermill-firestore.git" content/src-link/watermill-firestore
cloneOrPull "https://github.com/ThreeDotsLabs/watermill-bolt.git" content/src-link/watermill-bolt
cloneOrPull "https://github.com/ThreeDotsLabs/watermill-redisstream.git" content/src-link/watermill-redisstream
cloneOrPull "https://github.com/ThreeDotsLabs/watermill-aws.git" content/src-link/watermill-aws

find content/src-link -name '*.md' -delete
find content/src-link -name '*.html' -delete
Expand Down
9 changes: 0 additions & 9 deletions docs/content/pubsubs/amazonsqs.md
View file

This file was deleted.

248 changes: 248 additions & 0 deletions docs/content/pubsubs/aws.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
+++
title = "Amazon AWS SNS/SQS"
description = "AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple and scale microservices, distributed systems, and serverless applications."
date = 2024-10-19T15:30:00+02:00
bref = "AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple and scale microservices, distributed systems, and serverless applications."
weight = 10
+++

AWS SQS and SNS are fully-managed message queuing and Pub/Sub-like services that make it easy to decouple
and scale microservices, distributed systems, and serverless applications.

Watermill provides a simple way to use AWS SQS and SNS with Go.
It handles all the AWS SDK internals and provides a simple API to publish and subscribe messages.

Official Documentation:
- [SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html)
- [SNS](https://docs.aws.amazon.com/sns/latest/dg/welcome.html)

You can find a fully functional example with AWS SNS in the [Watermill examples](https://github.com/ThreeDotsLabs/watermill/tree/master/_examples/pubsubs/aws).

## Installation

```bash
go get github.com/ThreeDotsLabs/watermill-aws
```

## SQS vs SNS

While both SQS and SNS are messaging services provided by AWS, they serve different purposes and are best suited for different scenarios in your Watermill applications.

### How SNS is connected with SQS

To use SNS as a Pub/Sub (to have multiple subscribers receiving the same message), you need to create an SNS topic and subscribe to SQS queues.
When a message is published to the SNS topic, it will be delivered to all subscribed SQS queues.
We implemented this logic in the `watermill-aws` package out of the box.

When you subscribe to an SNS topic, Watermill AWS creates an SQS queue and subscribes to it.

[![](https://mermaid.ink/img/pako:eNptkU1uwyAQRq-CWLWScwEW2TjbVnboru4Cw9hB5ccZoFIV5e7FxVRKUjaM5j0-jZgLlV4BZTTAOYGTcNBiRmEHR_JZBEYt9SJcJB0RgfBXTro0Gh1OgI_OijfrzS9a_mP0xchXnyABeXcfj1ZbU3gag0Q9AhaxqN1uv8-U1VGIhRDEDKHgjFahzwIHpyot0Hi_lKqtUueNIZPH-5ie77LSMnKEmNDd5rQFdehlblf2FJ7vwg9gIMLt2zV5w0ew_usPkwm9Jef1Y4qZx6cNtYBWaJW3dFnbA40nsDBQlksl8HOgg7tmT6To-beTlEVM0NC0KBHrRimbhAm5C0pHjy9l7b_bbyj6NJ824_oDU0CtBA?type=png)](https://mermaid.live/edit#pako:eNptkU1uwyAQRq-CWLWScwEW2TjbVnboru4Cw9hB5ccZoFIV5e7FxVRKUjaM5j0-jZgLlV4BZTTAOYGTcNBiRmEHR_JZBEYt9SJcJB0RgfBXTro0Gh1OgI_OijfrzS9a_mP0xchXnyABeXcfj1ZbU3gag0Q9AhaxqN1uv8-U1VGIhRDEDKHgjFahzwIHpyot0Hi_lKqtUueNIZPH-5ie77LSMnKEmNDd5rQFdehlblf2FJ7vwg9gIMLt2zV5w0ew_usPkwm9Jef1Y4qZx6cNtYBWaJW3dFnbA40nsDBQlksl8HOgg7tmT6To-beTlEVM0NC0KBHrRimbhAm5C0pHjy9l7b_bbyj6NJ824_oDU0CtBA)

We can say, that a single SQS queue acts as a consumer group or subscription in other Pub/Sub implementations.

The mechanism is detailed in [AWS documentation](https://docs.aws.amazon.com/sns/latest/dg/subscribe-sqs-queue-to-sns-topic.html).

### How to choose between SQS and SNS

#### SQS (Simple Queue Service)

- Use when you need a simple message queue with a single consumer.
- Great for task queues or background job processing.
- Supports exactly-once processing (with FIFO queues) and guaranteed order (mostly).

Example use case: Processing user uploads in the background.

[![](https://mermaid.ink/img/pako:eNplkT1uwzAMRq8icGoB5wIasjhrASdevdASHQvVj0NJAYogd69c2wmaaCL4HilI3w1U0AQSIl0yeUUHg2dG13lRzoScjDIT-iQagVG0x1Y0ubcmjsTvzoxX65gp07tRb7zNfVRs-nnNojW7_b4QuV0gHMWIZ4oLtiFMS1U_xGCtGAK_mIXtilJLcaKU2W_4OV1Qw0GV9sY-4ufL8gNZSvR_dt684hO5cH1gMXBw4vJ8M3kNFThih0aX773N7Q7SSI46kKXUyN8ddP5ePMwptD9egUycqYI8aUxbFCAHtLF0SZsU-GvJ6y-2Cjjk87ga91_dnpaW?type=png)](https://mermaid.live/edit#pako:eNplkT1uwzAMRq8icGoB5wIasjhrASdevdASHQvVj0NJAYogd69c2wmaaCL4HilI3w1U0AQSIl0yeUUHg2dG13lRzoScjDIT-iQagVG0x1Y0ubcmjsTvzoxX65gp07tRb7zNfVRs-nnNojW7_b4QuV0gHMWIZ4oLtiFMS1U_xGCtGAK_mIXtilJLcaKU2W_4OV1Qw0GV9sY-4ufL8gNZSvR_dt684hO5cH1gMXBw4vJ8M3kNFThih0aX773N7Q7SSI46kKXUyN8ddP5ePMwptD9egUycqYI8aUxbFCAHtLF0SZsU-GvJ6y-2Cjjk87ga91_dnpaW)

#### SNS (Simple Notification Service)

- Use when you need to broadcast messages to multiple subscribers.
- Perfect for implementing pub/sub patterns.
- Useful for event-driven architectures.
- Supports multiple types of subscribers (SQS, Lambda, HTTP/S, email, SMS, etc.).

Example use case: Notifying multiple services about a new user registration.

Our SNS implementation in Watermill automatically creates and manages SQS queues for each subscriber, simplifying the process of using SNS with multiple SQS queues.

Remember, you can use both in the same application where appropriate. For instance, you might use SNS to broadcast events and SQS to process specific tasks triggered by those events.

To learn how SNS and SQS work together, see the [How SNS is connected with SQS](#how-sns-is-connected-with-sqs) section.

## SQS

### Characteristics

| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | no | it's a queue, for consumer groups-like functionality use [SNS](#sns) |
| ExactlyOnceDelivery | no | [yes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) |
| GuaranteedOrder | yes\* | from [AWS Docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html): _"(...) due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order. Despite this, standard queues make a best-effort attempt to maintain the order in which messages are sent."_ |
| Persistent | yes | |

### Required permissions

- `"sqs:ReceiveMessage"`
- `"sqs:DeleteMessage"`
- `"sqs:GetQueueUrl"`
- `"sqs:CreateQueue"`
- `"sqs:GetQueueAttributes"`
- `"sqs:SendMessage"`
- `"sqs:ChangeMessageVisibility"`

[todo - verify]

### SQS Configuration

{{% load-snippet-partial file="src-link/watermill-aws/sqs/config.go" first_line_contains="type SubscriberConfig struct " last_line_contains="type GenerateCreateQueueInputFunc" %}}

### Resolving Queue URL

In the Watermill model, we are normalizing the AWS queue url to `topic` used in the `Publish` and `Subscribe` methods.

To give you flexibility of what you want to use as a topic in Watermill, you can customize resolving the queue URL.

{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// QueueUrlResolver" last_line_contains="GenerateQueueUrlResolver" %}}

You can implement your own `QueueUrlResolver` or use one of the provided resolvers.

By default, `GetQueueUrlByNameUrlResolver` resolver is used:

{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// GetQueueUrlByNameUrlResolver " last_line_contains="NewGetQueueUrlByNameUrlResolver" %}}

There are two more resolvers available:

{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// GenerateQueueUrlResolver" last_line_contains="}" %}}

{{% load-snippet-partial file="src-link/watermill-aws/sqs/url_resolver.go" first_line_contains="// TransparentUrlResolver" last_line_contains="}" %}}

### Using with SQS emulator

You may want to use [`goaws`](https://github.com/Admiral-Piett/goaws) or [`localstack`](https://hub.docker.com/r/localstack/localstack) for local development or testing.

You can override the endpoint using the `OptFns` option in the `SubscriberConfig` or `PublisherConfig`.

```go
package main

import (
amazonsqs "github.com/aws/aws-sdk-go-v2/service/sqs"
"github.com/ThreeDotsLabs/watermill-amazonsqs/sqs"
)

func main() {
// ...

sqsOpts := []func(*amazonsqs.Options){
amazonsqs.WithEndpointResolverV2(sqs.OverrideEndpointResolver{
Endpoint: transport.Endpoint{
URI: *lo.Must(url.Parse("http://localstack:4566")),
},
}),
}

sqsConfig := sqs.SubscriberConfig{
AWSConfig: cfg,
OptFns: sqsOpts,
}

sub, err := sqs.NewSubscriber(sqsConfig, logger)
if err != nil {
panic(fmt.Errorf("unable to create new subscriber: %w", err))
}

// ...
}

```

## SNS

### Characteristics

| Feature | Implements | Note |
|---------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ConsumerGroups | yes | yes |
| ExactlyOnceDelivery | no | [yes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html) |
| GuaranteedOrder | yes\* | from [AWS Docs](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html): _"(...) due to the highly distributed architecture, more than one copy of a message might be delivered, and messages may occasionally arrive out of order. Despite this, standard queues make a best-effort attempt to maintain the order in which messages are sent."_ |
| Persistent | yes | |


### Required permissions

- `sns:Subscribe`
- `sns:ConfirmSubscription`
- `sns:Receive`
- `sns:Unsubscribe`

and all permissions required for SQS:

- `sqs:ReceiveMessage`
- `sqs:DeleteMessage`
- `sqs:GetQueueUrl`
- `sqs:CreateQueue`
- `sqs:GetQueueAttributes`
- `sqs:SendMessage`
- `sqs:ChangeMessageVisibility`
- `sqs:SetQueueAttributes`

Additionally, if `sns.SubscriberConfig.DoNotSetQueueAccessPolicy` is not enabled, you should have the following:

- `sqs:SetQueueAttributes`

### SNS Configuration

{{% load-snippet-partial file="src-link/watermill-aws/sns/config.go" first_line_contains="type SubscriberConfig struct " last_line_contains="type GenerateSqsQueueNameFn" %}}

Additionally, because SNS Subscriber uses SQS ques as "subscriptions", you need to pass [SQS configuration](#sqs-configuration) as well.

### Resolving Queue URL

In the Watermill model, we normalise AWS Topic ARN to the `topic` used in the `Publish` and `Subscribe` methods.

{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// TopicResolver" last_line_contains="}" %}}

We are providing two out-of-the-box resolvers:

{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// TransparentTopicResolver" last_line_contains="}" %}}

{{% load-snippet-partial file="src-link/watermill-aws/sns/topic.go" first_line_contains="// GenerateArnTopicResolver" last_line_contains="}" %}}

### Using with SNS emulator

You may want to use [`goaws`](https://github.com/Admiral-Piett/goaws) or [`localstack`](https://hub.docker.com/r/localstack/localstack) for local development or testing.

You can override the endpoint using the `OptFns` option in the `SubscriberConfig` or `PublisherConfig`.

```go
package main

import (
amazonsns "github.com/aws/aws-sdk-go-v2/service/sns"
"github.com/ThreeDotsLabs/watermill-amazonsns/sns"
)

func main() {
// ...

snsOpts := []func(*amazonsns.Options){
amazonsns.WithEndpointResolverV2(sns.OverrideEndpointResolver{
Endpoint: transport.Endpoint{
URI: *lo.Must(url.Parse("http://localstack:4566")),
},
}),
}

snsConfig := sns.SubscriberConfig{
AWSConfig: cfg,
OptFns: snsOpts,
}

sub, err := sns.NewSubscriber(snsConfig, sqsConfig, logger)
if err != nil {
panic(fmt.Errorf("unable to create new subscriber: %w", err))
}

// ...
}
```
2 changes: 1 addition & 1 deletion docs/resources/_gen/assets/scss/app.scss_cdf9d7c9eb97e4550ded64a8776dd9e8.content
View file

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/resources/_gen/assets/scss/app.scss_cdf9d7c9eb97e4550ded64a8776dd9e8.json
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"Target":"main.e18d306ea87fc31ccdc18bc02f9617933c7a3056a2ba2de6830a8ba47283543cbbd494e24b7d778b41d54d3074f438879353bddabcc0bd0d0389eb3d84f0aa38.css","MediaType":"text/css","Data":{"Integrity":"sha512-4Y0wbqh/wxzNwYvAL5YXkzx6MFaiui3mgwqLpHKDVDy71JTiS313i0HVTTB09DiHk1O92rzAvQ0Dies9hPCqOA=="}}
{"Target":"main.3e9cfa18ef8eea818e5322c5eccce2bcedbda0100f6cc0469a0d0d315f86d01a5e46c2424f782e7e0c2e12da5c5afc98309d50b5aa60f98949293da86193c731.css","MediaType":"text/css","Data":{"Integrity":"sha512-Ppz6GO+O6oGOUyLF7MzivO29oBAPbMBGmg0NMV+G0BpeRsJCT3gufgwuEtpcWvyYMJ1Qtapg+YlJKT2oYZPHMQ=="}}
5 changes: 5 additions & 0 deletions netlify.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,8 @@
from = "/docs/pub-sub-implementing"
to = "/development/pub-sub-implementing/"
status = 301

[[redirects]]
from = "/pubsubs/amazonsqs/"
to = "/pubsubs/aws/"
status = 301

0 comments on commit 94c0209

Please sign in to comment.