-
Notifications
You must be signed in to change notification settings - Fork 482
docs: tweaks to include subscribe based sinks #33792
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,14 +12,15 @@ aliases: | |
|
|
||
| ## Overview | ||
|
|
||
| Sinks are the inverse of sources and represent a connection to an external stream | ||
| where Materialize outputs data. When a user defines a sink over a materialized view, | ||
| source, or table, Materialize automatically generates the required schema and writes down | ||
| the stream of changes to that view or source. In effect, Materialize sinks act as | ||
| change data capture (CDC) producers for the given source or view. | ||
| Sinks are the inverse of sources and represent a connection to an external | ||
| stream where Materialize outputs data. You can sink data from a **materialized** | ||
| view, a source, or a table. | ||
|
|
||
| Currently, Materialize only supports sending sink data to Kafka. See | ||
| the [Kafka sink documentation](/sql/create-sink/kafka) for details. | ||
| ## Sink methods | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
|
||
| To create a sink, you can: | ||
|
|
||
| {{< yaml-table data="sink_external_systems" >}} | ||
|
|
||
| ## Clusters and sinks | ||
|
|
||
|
|
@@ -29,7 +30,8 @@ See also [Operational guidelines](/manage/operational-guidelines/). | |
|
|
||
| ## Hydration considerations | ||
|
|
||
| During creation, sinks need to load an entire snapshot of the data in memory. | ||
| During creation, Kafka sinks need to load an entire snapshot of the data in | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Probably will tweak to Kakfa/Redpanda sinks since ... |
||
| memory. | ||
|
|
||
| ## Related pages | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,31 +11,20 @@ menu: | |
| --- | ||
|
|
||
| A [sink](/concepts/sinks/) describes the external system you want Materialize to | ||
| write data to and details the encoding of that data. | ||
| write data to and details the encoding of that data. You can sink data from a | ||
| **materialized** view, a source, or a table. | ||
|
|
||
| ### Creating a sink | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moved into the kafka page since kafka specific. |
||
| ## Sink methods | ||
|
|
||
| When a user defines a sink over a **materialized** view, source, or table, | ||
| Materialize automatically generates the required schema and writes down the | ||
| stream of changes to that view or source. In effect, Materialize sinks act as | ||
| change data capture (CDC) producers for the given source or view. | ||
| To create a sink, you can: | ||
|
|
||
| During creation, sinks need to load an entire snapshot of the data in memory. | ||
| {{< yaml-table data="sink_external_systems" >}} | ||
|
|
||
| ### Operational guideline | ||
|
|
||
| - Avoid putting sinks on the same cluster that hosts sources to allow for | ||
| [blue/green deployment](/manage/dbt/blue-green-deployments). | ||
|
|
||
| ### Available guides | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Included in the table. |
||
|
|
||
| The following guides are available for sinking results from Materialize to external systems: | ||
|
|
||
| - [Sinking results to Amazon S3](/serve-results/sink/s3/) | ||
| - [Sinking results to Census](/serve-results/sink/census/) | ||
| - [Sinking results to Kafka/Redpanda](/serve-results/sink/kafka/) | ||
| - [Sinking results to Snowflake](/serve-results/sink/snowflake/) | ||
|
|
||
| ### Troubleshooting | ||
|
|
||
| For help, see [Troubleshooting | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,7 +15,10 @@ menu: | |
| ## Connectors | ||
|
|
||
| Materialize bundles a **native connector** that allow writing data to Kafka and | ||
| Redpanda. | ||
| Redpanda. When a user defines a sink to Kafka/Redpanda, Materialize | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Content ported over from Sinks pages since was specifically kafka/redpanda connector. |
||
| automatically generates the required schema and writes down the stream of | ||
| changes to that view or source. In effect, Materialize sinks act as change data | ||
| capture (CDC) producers for the given source or view. | ||
|
|
||
| For details on the connector, including syntax, supported formats and examples, | ||
| refer to [`CREATE SINK`](/sql/create-sink/kafka). | ||
|
|
@@ -24,10 +27,14 @@ refer to [`CREATE SINK`](/sql/create-sink/kafka). | |
|
|
||
| Redpanda uses the same syntax as Kafka [`CREATE SINK`](/sql/create-sink/kafka). | ||
|
|
||
| {{</ tip >}} | ||
| {{< /tip >}} | ||
|
|
||
| ## Features | ||
|
|
||
| ### Memory use during creation | ||
|
|
||
| During creation, sinks need to load an entire snapshot of the data in memory. | ||
|
|
||
| ### Automatic topic creation | ||
|
|
||
| If the specified Kafka topic does not exist, Materialize will attempt to create | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| columns: | ||
| - column: Method | ||
| - column: External system | ||
| - column: Guide(s) or Example(s) | ||
|
|
||
| rows: | ||
| - External system: "Amazon S3 or S3-compatible storage" | ||
| Method: "Use `COPY TO` command" | ||
| Guide(s) or Example(s): | | ||
| - [Sink to Amazon S3](/serve-results/sink/s3/) | ||
| - External system: "Census supported destinations" | ||
| Method: "Use Census as an intermediate step" | ||
| Guide(s) or Example(s): | | ||
| - [Sink to Census](/serve-results/sink/census/) | ||
| - External system: "Snowflake and other systems that can read from S3" | ||
| Method: "Use `COPY TO` S3 or S3-compatible storage as an intermediate step" | ||
| Guide(s) or Example(s): | | ||
| - [Sink to Snowflake](/serve-results/sink/snowflake/) | ||
| - External system: "Kafka/Redpanda" | ||
| Method: "Use a native connector" | ||
| Guide(s) or Example(s): | | ||
| - [Sink to Kafka/Redpanda](/serve-results/sink/kafka/) | ||
| - External system: "Various" | ||
| Method: "Use `SUBSCRIBE`" | ||
| Guide(s) or Example(s): | | ||
| - [Sink to Postgres](https://github.com/MaterializeInc/mz-catalog-sync) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should I mention dbmate in here? that is Subscribe + dbmate? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This repo is private 😅 I'll make it public. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh ... heh heh ... Thank you ❤️ ❤️ ❤️ There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just following up on this. Should I just ask operations to make the repo public? |
||
| - [Sink to Redis](https://github.com/MaterializeIncLabs/mz-redis-sync) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replaced kafka-specific content with more generic content (same as in the server-results/sink page)