demo-scene

Scripts and samples to support Confluent Platform talks. May be rough around the edges. For automated tutorials and QA'd code, see https://github.com/confluentinc/examples/

Requirements

You need to allocate Docker 8GB when running these. Avoid allocating all your machine's cores to Docker as this may cause the machine to become unresponsive when running large stacks. On a four-core Mac Book two cores for Docker should be fine.

Contents

• Data pipelines (Database + KSQL + Elasticsearch)

  • Pipeline to the cloud - on-premises RDBMS to Cloud datawarehouse e.g. Snowflake (✍️ blog)
  • Rail data streaming pipeline (🗣️talk)
  • Apache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline! (🗣️talk)
  • MySQL / Debezium CDC / KSQL / Elasticsearch
  • Oracle CDC / KSQL / Elasticsearch
  • Postgres / Debezium CDC / KSQL / Elasticsearch
  • CDC demo with MySQL (🗣️talk)
  • CDC demo with Oracle
  • Building data pipelines with Confluent Cloud and GCP (BigQuery, GCS, etc)

• ksqlDB (previously known as KSQL)

  • Introduction to ksqlDB 01 (🗣️talk)
  • Introduction to KSQL 02
  • Using Twitter data with ksqlDB
  • KSQL UDF Advanced Example
  • KSQL Troubleshooting
  • ATM Fraud detection with Kafka and KSQL (🗣️talk)
  • Kafka Streams/KSQL Movie Demo
  • KSQL MQTT demo
  • KSQL Dump Utility
  • KSQL workshop (more recent version is here)
  • Multi-node ksqlDB and Kafka Connect clusters

• Kafka Connect

  • From Zero to Hero with Kafka Connect (🗣️talk)
  • Kafka Connect Converters and Serialization (✍️ blog)
  • Building a Kafka Connect cluster
  • Kafka Connect error handling (✍️ blog)
  • Multi-node ksqlDB and Kafka Connect clusters
  • Specific connectors
    • 👉 S3 Sink (🎥 tutorial)
    • 👉 Database (tutorial 🎥 1 / 🎥 2)
    • 👉 Elasticsearch (🎥 Tutorial)
    • Kafka Connect JDBC Source demo environment (✍️ blog)
    • InfluxDB & Kafka Connect (✍️ blog)
    • RabbitMQ into Kafka (✍️ blog)
    • MQTT Connect Connector Demo
    • Example Kafka Connect syslog configuration and Docker Compose (see blog series 1/2/3 and standalone articles here and here)
    • Azure SQL Data Warehouse Connector Sink Demo
    • IBM MQ Connect Connector Demo
    • Solace Sink/Source Demo
    • RSS feed into Kafka

• Confluent Cloud

  • Pac-Man Demo
  • "The Cube" Demo
  • Using Replicator with Confluent Cloud

• Misc

  • Hacky export/import between Kafka clusters using kafkacat
  • Docker Compose for just the community licensed components of Confluent Platform
  • Topic Tailer, stream topics to the browser using websockets
  • KPay payment processing example
  • Industry themes (e.g. banking Next Best Offer)
  • Tiered Storage Demo
  • Distributed tracing
  • Analysing Sonos data in Kafka (✍️ blog)
  • Analysing Wi-Fi pcap data with Kafka
  • Twitter streams and Operator

Feedback & Questions

Raise an issue on this github project, or head to http://cnfl.io/slack and join our Community Slack group.