Enable Snowflake Sink Connector to be able to consume Debezium events.

[airbots]

This is an interesting feature that I think very valuable. A lot of companies are using debezium source connector to drain data from databases to Kafka, but their debezium jdbc sink connector does not work with snowflake since the snowflake does not support JDBC dialect.

If snowflake sink connector can handle the events that debezium send to kafka and deal with update, insert, and delete. It will close a loop and will be a very powerful tool.

[sfc-gh-wfateem]

Hi @airbots,

Thanks for your comments. The Snowflake Kafka Connector is just a sink connector, so the data residing in Kafka that we're reading from the broker and sending to Snowflake is source-agnostic. It shouldn't matter what your source connector is (Debezium) and what the events are, those messages are going to be serialized using some converter (Avro, JSON, etc...).

I think you're asking for supportability for Snowflake as a destination for the Debezium JDBC sink connector. JDBC is just a specification, but each database will have its own dialect. For example, the Debezium JDBC sink connector has different modes for inserting data, and that's configurable using the parameter insert.mode. One of those modes is just a plain INSERT. Now depending on the destination database, it's going to be the Debezium JDBC sink connector's job to translate the action it wants to do into the appropriate dialect to ensure that the INSERT statement it issues is syntactically correct for Oracle, Postgres, or MySQL. The Debezium connector doesn't have support for the Snowflake dialect. Your issues isn't going to be with that insert mode, I don't believe, because the INSERT statements across those databases and Snowflake are very similar. Your issue is going to be with upserts because for the longest time, Snowflake did not enforce primary keys. This has changed for Hybrid tables.

So I imagine you're asking us to either:

1.) Work with Debezium to add support for Snowflake in the JDBC sink connector.

OR

2.) Add a feature in the Snowflake Kafka Connector to support upserts, update, and delete operations based on the messages we read from the Kafka topic.

For (1), I'm not sure if we could influence Debezium to add support for that. I recommend you bring this up with the account team.

For (2), I see other users have asked for this feature already in #505. If this is ultimately what you're asking for, then I recommend you post a comment there to show that same interest, and I can bring it up with the team to review.

[sfc-gh-wfateem]

Hey @airbots,

Do you have any thoughts on my previous comment?

[airbots]

HI Mr. Waleed, thank you for the suggestions. I mean the #505 . I am also working on a local version, it possible, I can submit my changes to a PR and all can review and provide suggestions.

[sfc-gh-wfateem]

Hi @airbots,

Thanks for confirming. Given that your request is a duplicate of #505 I'll go ahead and close off this issue.