Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions doc/user/content/concepts/sinks.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,15 @@ aliases:

## Overview

Sinks are the inverse of sources and represent a connection to an external stream
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced kafka-specific content with more generic content (same as in the server-results/sink page)

where Materialize outputs data. When a user defines a sink over a materialized view,
source, or table, Materialize automatically generates the required schema and writes down
the stream of changes to that view or source. In effect, Materialize sinks act as
change data capture (CDC) producers for the given source or view.
Sinks are the inverse of sources and represent a connection to an external
stream where Materialize outputs data. You can sink data from a **materialized**
view, a source, or a table.

Currently, Materialize only supports sending sink data to Kafka. See
the [Kafka sink documentation](/sql/create-sink/kafka) for details.
## Sink methods
Copy link
Contributor Author

@kay-kim kay-kim Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


To create a sink, you can:

{{< yaml-table data="sink_external_systems" >}}

## Clusters and sinks

Expand All @@ -29,7 +30,8 @@ See also [Operational guidelines](/manage/operational-guidelines/).

## Hydration considerations

During creation, sinks need to load an entire snapshot of the data in memory.
During creation, Kafka sinks need to load an entire snapshot of the data in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably will tweak to Kakfa/Redpanda sinks since ...

memory.

## Related pages

Expand Down
21 changes: 5 additions & 16 deletions doc/user/content/serve-results/sink/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,31 +11,20 @@ menu:
---

A [sink](/concepts/sinks/) describes the external system you want Materialize to
write data to and details the encoding of that data.
write data to and details the encoding of that data. You can sink data from a
**materialized** view, a source, or a table.

### Creating a sink
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved into the kafka page since kafka specific.

## Sink methods

When a user defines a sink over a **materialized** view, source, or table,
Materialize automatically generates the required schema and writes down the
stream of changes to that view or source. In effect, Materialize sinks act as
change data capture (CDC) producers for the given source or view.
To create a sink, you can:

During creation, sinks need to load an entire snapshot of the data in memory.
{{< yaml-table data="sink_external_systems" >}}

### Operational guideline

- Avoid putting sinks on the same cluster that hosts sources to allow for
[blue/green deployment](/manage/dbt/blue-green-deployments).

### Available guides
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Included in the table.


The following guides are available for sinking results from Materialize to external systems:

- [Sinking results to Amazon S3](/serve-results/sink/s3/)
- [Sinking results to Census](/serve-results/sink/census/)
- [Sinking results to Kafka/Redpanda](/serve-results/sink/kafka/)
- [Sinking results to Snowflake](/serve-results/sink/snowflake/)

### Troubleshooting

For help, see [Troubleshooting
Expand Down
11 changes: 9 additions & 2 deletions doc/user/content/serve-results/sink/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ menu:
## Connectors

Materialize bundles a **native connector** that allow writing data to Kafka and
Redpanda.
Redpanda. When a user defines a sink to Kafka/Redpanda, Materialize
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Content ported over from Sinks pages since was specifically kafka/redpanda connector.

automatically generates the required schema and writes down the stream of
changes to that view or source. In effect, Materialize sinks act as change data
capture (CDC) producers for the given source or view.

For details on the connector, including syntax, supported formats and examples,
refer to [`CREATE SINK`](/sql/create-sink/kafka).
Expand All @@ -24,10 +27,14 @@ refer to [`CREATE SINK`](/sql/create-sink/kafka).

Redpanda uses the same syntax as Kafka [`CREATE SINK`](/sql/create-sink/kafka).

{{</ tip >}}
{{< /tip >}}

## Features

### Memory use during creation

During creation, sinks need to load an entire snapshot of the data in memory.

### Automatic topic creation

If the specified Kafka topic does not exist, Materialize will attempt to create
Expand Down
31 changes: 31 additions & 0 deletions doc/user/data/sink_external_systems.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
columns:
- column: Method
- column: External system
- column: Guide(s) or Example(s)

rows:
- External system: "Amazon S3 or S3-compatible storage"
Method: "Use `COPY TO` command"
Guide(s) or Example(s): |
- [Sink to Amazon S3](/serve-results/sink/s3/)
- External system: "Census supported destinations"
Method: "Use Census as an intermediate step"
Guide(s) or Example(s): |
- [Sink to Census](/serve-results/sink/census/)
- External system: "Snowflake and other systems that can read from S3"
Method: "Use `COPY TO` S3 or S3-compatible storage as an intermediate step"
Guide(s) or Example(s): |
- [Sink to Snowflake](/serve-results/sink/snowflake/)
- External system: "Kafka/Redpanda"
Method: "Use a native connector"
Guide(s) or Example(s): |
- [Sink to Kafka/Redpanda](/serve-results/sink/kafka/)
- External system: "Various"
Method: "Use `SUBSCRIBE`"
Guide(s) or Example(s): |
- [Sink to Postgres](https://github.com/MaterializeInc/mz-catalog-sync)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I mention dbmate in here? that is Subscribe + dbmate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This repo is private 😅 I'll make it public.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh ... heh heh ... Thank you ❤️ ❤️ ❤️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just following up on this. Should I just ask operations to make the repo public?

- [Sink to Redis](https://github.com/MaterializeIncLabs/mz-redis-sync)