diff --git a/src/assets/images/user-guide/integrations/kafka-integration-architecture.svg b/src/assets/images/user-guide/integrations/kafka-integration-architecture.svg new file mode 100644 index 0000000000..d3770e51d9 --- /dev/null +++ b/src/assets/images/user-guide/integrations/kafka-integration-architecture.svg @@ -0,0 +1,154 @@ + + + + + + + + + + + + + + + + + + ThingsBoard Deployment + + + + + + + + + + + + + + + + + + + + Existing Device + + + + + Produce + messages / + Publish + messages + + + + Receive + commands + + + + kafka + + + + + + + + Consume + messages / + Poll messages + + + + Kafka + Integration + + + + + + + + + uplink payload + + + + Data Converter + + + + + + Uplink + + + + + telemetry / + attributes + + + + ThingsBoard + Core Service + + + + + + + + + + Storage + + + + + + + + + + + Dashboards + + + + + + + + + + + + + + + Rule Engine + + + + + + Produce commands / Publish commands + + + + + Uplink — data from Kafka to ThingsBoard + + + Downlink — Rule Node publishes back to Kafka + + + + diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/create_integration/kafka_integration_properties_1.png b/src/assets/images/user-guide/integrations/kafka/confluent/create_integration/kafka_integration_properties_1.png deleted file mode 100644 index 51e8f1babf..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/create_integration/kafka_integration_properties_1.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/create_integration/kafka_work_with_cluster_1.png b/src/assets/images/user-guide/integrations/kafka/confluent/create_integration/kafka_work_with_cluster_1.png deleted file mode 100644 index 067e82d3db..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/create_integration/kafka_work_with_cluster_1.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_0.png b/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_0.png deleted file mode 100644 index 98b3a11238..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_0.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_1.png b/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_1.png deleted file mode 100644 index 2ea0f49d73..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_1.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_2.png b/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_2.png deleted file mode 100644 index b98bce1a4f..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_node_2.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_result_0.png b/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_result_0.png deleted file mode 100644 index d8c4070557..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_result_0.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_result_1.png b/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_result_1.png deleted file mode 100644 index d2292cd39e..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/confluent/downlink/kafka_confluent_downlink_result_1.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-1.png index 5db765e5ba..a88e462299 100644 Binary files a/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-1.png and b/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-2.png b/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-2.png index 3be1928ffe..02cf5d9231 100644 Binary files a/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-2.png and b/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-2.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-3.png b/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-3.png index db6a0db7f3..c1aaf98e99 100644 Binary files a/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-3.png and b/src/assets/images/user-guide/integrations/kafka/kafka-add-integration-3.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-1.png new file mode 100644 index 0000000000..16be267d89 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-2.png b/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-2.png new file mode 100644 index 0000000000..6e329f1547 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-2.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-3.png b/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-3.png new file mode 100644 index 0000000000..41cc83d3ac Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-configure-rule-chain-3.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-create-downlink-topic-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-create-downlink-topic-1.png new file mode 100644 index 0000000000..58d9d2e4ab Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-create-downlink-topic-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-1.png new file mode 100644 index 0000000000..13b4287341 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-2.png b/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-2.png new file mode 100644 index 0000000000..706c82a8c2 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-2.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-3.png b/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-3.png new file mode 100644 index 0000000000..e73163e0f1 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-downlink-node-events-3.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-produce-test-record-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-produce-test-record-1.png new file mode 100644 index 0000000000..43122ef14e Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-produce-test-record-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-produce-test-record-2.png b/src/assets/images/user-guide/integrations/kafka/kafka-produce-test-record-2.png new file mode 100644 index 0000000000..5d6dcaf781 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-produce-test-record-2.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-start-downlink-consumer-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-start-downlink-consumer-1.png new file mode 100644 index 0000000000..2041711224 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-start-downlink-consumer-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-trigger-downlink-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-trigger-downlink-1.png new file mode 100644 index 0000000000..f638cd79f1 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-trigger-downlink-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-1.png new file mode 100644 index 0000000000..40bb7d3f08 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-2.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-2.png new file mode 100644 index 0000000000..74336da7fc Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-2.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-3.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-3.png new file mode 100644 index 0000000000..e3e0de1751 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-converter-events-3.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-device-provisioning-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-device-provisioning-1.png new file mode 100644 index 0000000000..e8ce4fdd7a Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-device-provisioning-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-integration-events-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-integration-events-1.png new file mode 100644 index 0000000000..0bc939e8e1 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-integration-events-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-integration-events-2.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-integration-events-2.png new file mode 100644 index 0000000000..37d65484a5 Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-integration-events-2.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka-verify-kafka-delivery-1.png b/src/assets/images/user-guide/integrations/kafka/kafka-verify-kafka-delivery-1.png new file mode 100644 index 0000000000..a1862425fa Binary files /dev/null and b/src/assets/images/user-guide/integrations/kafka/kafka-verify-kafka-delivery-1.png differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka_integration_test_send_msg_result.png b/src/assets/images/user-guide/integrations/kafka/kafka_integration_test_send_msg_result.png deleted file mode 100644 index 2d29ba211b..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/kafka_integration_test_send_msg_result.png and /dev/null differ diff --git a/src/assets/images/user-guide/integrations/kafka/kafka_uplink_test_decoder.png b/src/assets/images/user-guide/integrations/kafka/kafka_uplink_test_decoder.png deleted file mode 100644 index 9fb43f101c..0000000000 Binary files a/src/assets/images/user-guide/integrations/kafka/kafka_uplink_test_decoder.png and /dev/null differ diff --git a/src/content/_includes/docs/user-guide/integrations/kafka.mdx b/src/content/_includes/docs/user-guide/integrations/kafka.mdx index 08634a84e7..4387f30974 100644 --- a/src/content/_includes/docs/user-guide/integrations/kafka.mdx +++ b/src/content/_includes/docs/user-guide/integrations/kafka.mdx @@ -7,356 +7,691 @@ import { Products } from '~/models/site.models'; import Tabs from '@components/Tabs.astro'; import TabItem from '@components/TabItem.astro'; - + +The Apache Kafka integration connects ThingsBoard to Apache Kafka message brokers. Use it to consume records that producers — devices, IoT platforms, connectivity providers, or your own applications — publish to Kafka topics, and bring them into ThingsBoard as telemetry and attributes for monitoring and rule-based automation — including publishing commands back to Kafka. -Kafka Integration connects ThingsBoard to Apache Kafka message brokers. It subscribes to one or more Kafka topics, consumes incoming messages, and transforms them into ThingsBoard telemetry and attributes via an uplink converter. +## Architecture - + + +A producer publishes a record to a Kafka topic. The Kafka Integration, acting as a Kafka consumer, subscribes to the topic and polls the broker for new records. The uplink converter decodes the record value into telemetry and attributes, and the ThingsBoard Core Service stores the data, creating the device automatically on first contact. For downlink, the Rule Engine hands a message to a Kafka Rule Node, which publishes it to a separate Kafka topic; external consumers then read it. The Kafka Integration has no dedicated downlink converter. ## Prerequisites -A running Kafka broker is required before creating the integration. +Before creating the integration, ensure: -Use a local installation or a cloud-managed cluster such as Confluent Cloud. +- You have access to {props.product !== Products.PAAS && props.product !== Products.PAAS_EU && <>ThingsBoard PE or }ThingsBoard Cloud with integration functionality enabled for your tenant. +- You have permissions to create integrations and data converters. +- You have a running Apache Kafka broker — either a local installation or a cloud-managed cluster such as Confluent Cloud. If you do not have a cluster yet, follow the **[Kafka Setup](#kafka-setup)** step below. -## ThingsBoard integration setup + -### Create an uplink converter +## Kafka Setup -The uplink converter decodes the Kafka message body and maps it to the ThingsBoard data model. Kafka uses a generic uplink converter. +The Kafka Integration connects ThingsBoard to an existing Apache Kafka broker as a consumer. The steps below stand up a minimal **single-node broker in KRaft mode (no ZooKeeper)**, which is enough to test the integration. If you use a managed cluster (Confluent Cloud, Amazon MSK, Aiven, Redpanda Cloud, and others), skip to [Required Connection Details](#required-connection-details) and obtain the values from your provider instead. -The decoder function receives: -- `payload` — raw bytes of the Kafka message value -- `metadata` — `integrationName` and any key-value pairs configured in the integration's Metadata settings +### Run a Single-Node Broker (KRaft) -It must return `deviceName`, `deviceType`, `telemetry`, and `attributes`. Both `telemetry` and `attributes` are flat key-value maps — nested objects are not supported. +KRaft is Kafka's built-in consensus protocol; since Kafka 4.0 it fully replaces ZooKeeper, so one process acts as both controller and broker. Kafka 4.x requires **Java 17+**, and formatting storage is a **required, explicit step** before the first start. The commands below were verified on Ubuntu with Kafka 4.3.1 — adjust the version and paths for your environment. -