Added profile clusterlinking in dev-toolkit - moved away from cp-demo - (#288)

* added profile clusterlinking in devtoolkit - moved away from cp-demo - #261

* added env on grafana dashboards

Author: hifly81

README.md

@@ -186,6 +186,7 @@ Currently supported profiles:
 - _consumer_: it will add a demo application implemented with Spring with full client metrics
 - _consumer-minimal_: it will add a demo application implemented with Spring with a limited number of client metrics
 - _jr_: it will add [JR](https://jrnd.io/) to generate random traffic for kafka.
+- _clusterlinking_: add Cluster Linking between _kafka1-kafka2-kafka3-kafka4_ and a new cluster with 1 broker _broker-dest_
 
 ## DEV-toolkit FAQ
 

dev-toolkit/docker-compose.clusterlinking.yaml

@@ -0,0 +1,38 @@
+services:
+  broker-replicator-dst:
+    image: confluentinc/cp-server:${CFLT_TAG}
+    hostname: broker-replicator-dst
+    container_name: broker-replicator-dst
+    profiles:
+      - clusterlinking
+    environment:
+      KAFKA_NODE_ID: 1
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
+      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://broker-replicator-dst:29092"
+      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_CONFLUENT_METADATA_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_CONFLUENT_SECURITY_EVENT_LOGGER_EXPORTER_KAFKA_TOPIC_REPLICAS: 1
+      KAFKA_CONFLUENT_CLUSTER_LINK_METADATA_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_CONFLUENT_REPORTER_RESULT_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_CONFLUENT_BALANCER_ENABLE: false
+      KAFKA_AUTO_CREATE_TOPICS: true
+      KAFKA_PROCESS_ROLES: "broker,controller"
+      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@broker-replicator-dst:29093"
+      KAFKA_LISTENERS: "PLAINTEXT://broker-replicator-dst:29092,CONTROLLER://broker-replicator-dst:29093"
+      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
+      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
+      KAFKA_LOG_DIRS: "/tmp/kraft-combined-logs"
+      KAFKA_LOG4J_LOGGERS: "org.apache.kafka.image.loader.MetadataLoader=WARN"
+      CLUSTER_ID: "N3Ept9vxTMijB5S74gkfPA"
+      KAFKA_CONFLUENT_HTTP_SERVER_LISTENERS: http://0.0.0.0:8090
+      KAFKA_REST_BOOTSTRAP_SERVERS: "broker-replicator-dst:29092"
+      KAFKA_OPTS: "-javaagent:/tmp/jmx_prometheus_javaagent-1.1.0.jar=1234:/tmp/kafka_config.yml"
+    volumes:
+      - $PWD/jmx-exporter/jmx_prometheus_javaagent-1.1.0.jar:/tmp/jmx_prometheus_javaagent-1.1.0.jar
+      - $PWD/jmx-exporter/kafka_broker.yml:/tmp/kafka_config.yml

dev-toolkit/start.sh

@@ -99,6 +99,23 @@ cat <<EOF >>assets/prometheus/prometheus-config/prometheus.yml
         replacement: '${1}'
 EOF
 
+# ADD Brokers dest cluster monitoring to prometheus config
+cat <<EOF >>assets/prometheus/prometheus-config/prometheus.yml
+
+  - job_name: "kafka-broker-dest"
+    static_configs:
+      - targets:
+          - "broker-replicator-dst:1234"
+        labels:
+          env: "dev"
+          job: "kafka-broker"
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: hostname
+        regex: '([^:]+)(:[0-9]+)?'
+        replacement: '${1}'
+EOF
+
 # ADD Schema Registry monitoring to prometheus config (default was for 1 SR only)
 cat <<EOF >>assets/prometheus/prometheus-config/prometheus.yml
 
@@ -128,6 +145,7 @@ $DOCKER_COMPOSE_CMD ${docker_args[@]} \
   -f docker-compose.consumer-minimal.yaml \
   -f docker-compose.schema-registry-primary-secondary.yaml \
   -f docker-compose.jr.yaml \
+  -f docker-compose.clusterlinking.yaml \
   up -d
 
 # if docker_args contains replicator, then start the replicator
@@ -168,6 +186,28 @@ if [[ " ${docker_args[@]} " =~ " ksqldb " ]]; then
 
 fi
 
+# if docker_args contains clusterlinking, then start the clusterlinking
+if [[ " ${docker_args[@]} " =~ " clusterlinking " ]]; then
+
+  echo -e "\nWaiting 45 seconds before starting clusterlinking..."
+  sleep 45
+
+  docker exec kafka1 bash -c "KAFKA_OPTS= kafka-topics --bootstrap-server kafka1:29092 --create --topic product --replication-factor 1 --partitions 1"
+
+  echo -e "Create link main-to-disaster-cl"
+
+  docker exec broker-replicator-dst bash -c '\
+  echo "\
+  bootstrap.servers=kafka1:29092
+  " > /home/appuser/cl.properties'
+
+  docker exec broker-replicator-dst bash -c "KAFKA_OPTS= kafka-cluster-links --bootstrap-server broker-replicator-dst:29092 --create --link main-to-disaster-cl --config-file /home/appuser/cl.properties"
+
+  docker exec broker-replicator-dst bash -c "KAFKA_OPTS= kafka-mirrors --create --source-topic product --mirror-topic product --link main-to-disaster-cl --bootstrap-server broker-replicator-dst:29092"
+
+
+fi
+
 echo -e "\ndev-toolkit started!"
 
 # Look at Prometheus metrics

dev-toolkit/stop.sh

@@ -25,6 +25,7 @@ $DOCKER_COMPOSE_CMD \
     -f docker-compose.consumer-minimal.yaml \
     -f docker-compose.schema-registry-primary-secondary.yaml \
     -f docker-compose.jr.yaml \
+    -f docker-compose.clusterlinking.yaml \
     down -v
 rm -rf jmx-exporter
 rm -rf assets

jmxexporter-prometheus-grafana/README.md

@@ -18,30 +18,30 @@ After run the demo, for Grafana, go to http://localhost:3000 and then login with
 
 List of provided dashboards:
 
- - [Confluent Platform overview](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#confluent-platform-overview)
- - [Zookeeper cluster](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#zookeeper-cluster)
- - [Kafka cluster](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-cluster)
- - [Kafka topics](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-topics)
- - [Kafka clients](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-clients)
- - [Kafka quotas](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-quotas)
- - [Kafka lag exporter](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-lag-exporter)
- - [Kafka transaction coordinator](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-transaction-coordinator)
- - [Schema Registry cluster](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#schema-registry-cluster)
- - [Kafka Connect cluster](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-connect-cluster)
- - [ksqlDB cluster](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#ksqldb-cluster)
- - [Kafka streams](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-streams)
- - [Kafka streams RocksDB](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-streams-rocksdb)
- - [Librdkafka based client](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#Librdkafka)
+ - [Confluent Platform overview _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#confluent-platform-overview)
+ - [Zookeeper cluster _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#zookeeper-cluster)
+ - [Kafka cluster _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-cluster)
+ - [Kafka topics _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-topics)
+ - [Kafka clients _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-clients)
+ - [Kafka quotas _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-quotas)
+ - [Kafka lag exporter _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-lag-exporter)
+ - [Kafka transaction coordinator _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-transaction-coordinator)
+ - [Schema Registry cluster _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#schema-registry-cluster)
+ - [Kafka Connect cluster _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-connect-cluster)
+ - [ksqlDB cluster _(cp-demo,dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#ksqldb-cluster)
+ - [Kafka streams _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-streams)
+ - [Kafka streams RocksDB _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kafka-streams-rocksdb)
+ - [Librdkafka based client _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#Librdkafka)
  - [Oracle CDC source Connector](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#oracle-cdc-source-connector)
  - [Debezium source Connectors](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#debezium-cdc-source-connectors)
  - [Mongo source and sink Connectors](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#mongo-source-and-sink-connectors)
- - [Cluster Linking](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#cluster-linking)
- - [Rest Proxy](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#rest-proxy)
- - [KRaft overview](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kraft)
- - [Confluent RBAC](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#rbac)
- - [Replicator](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#replicator)
+ - [Cluster Linking _(dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#cluster-linking)
+ - [Rest Proxy _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#rest-proxy)
+ - [KRaft overview _(dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#kraft)
+ - [Confluent RBAC _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#rbac)
+ - [Replicator _(dev-toolkit)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#replicator)
  - [Tiered Storage](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#tiered-storage)
- - [Confluent Audit](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#confluent-audit)
+ - [Confluent Audit _(cp-demo)_](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#confluent-audit)
  - [Flink Cluster](https://github.com/confluentinc/jmx-monitoring-stacks/blob/main/jmxexporter-prometheus-grafana/README.md#flink-cluster)
 
 ---
@@ -174,29 +174,13 @@ To test:
 
 ### Cluster Linking
 
-Demo is based on https://github.com/confluentinc/demo-scene/tree/master/cluster-linking-disaster-recovery
+To test use dev-toolkit with _clusterlinking_ profile:
 
-To test follow the next steps:
-
-1. Set env:
- ```bash
-MONITORING_STACK=jmxexporter-prometheus-grafana
- ```
-2. Clone demo cluster linking disaster recovery from confluentinc/demo-scene:
-```bash
-   [[ -d "clink-demo" ]] || git clone [email protected]:confluentinc/demo-scene.git clink-demo
-   (cd clink-demo && git fetch && git pull)
-   ```
-3. Start the monitoring solution with the STACK selected. This command also starts clink-demo, you do not need to start clink-demo separately.
-
-```bash
-${MONITORING_STACK}/cluster-linking/start.sh
-``` 
-
-4. Stop the monitoring solution. This command also stops clink-demo, you do not need to stop clink-demo separately.
+1. Start dev-toolkit with
 
 ```bash
-${MONITORING_STACK}/cluster-linking/stop.sh
+$ cd dev-toolkit
+$ start.sh --profile clusterlinking
 ```
 
 ![clusterlinking](img/clusterlinking.png)
@@ -208,7 +192,7 @@ ${MONITORING_STACK}/cluster-linking/stop.sh
 
 ### KRaft
 
-To test use dev-toolkit with _Default_ profile:
+To test use dev-toolkit with _default_ profile:
 
 1. Start dev-toolkit with