Final Kafka Tutorial (#109)

Co-authored-by: evyatarmeged <[email protected]> Co-authored-by: Uri Sarid <[email protected]> Co-authored-by: Tomer Greenwald <[email protected]> Co-authored-by: Ori Shoshan <[email protected]> Co-authored-by: omris94 <[email protected]> Co-authored-by: otterizebot <[email protected]>
otterize · Aug 30, 2023 · ea7e72b · ea7e72b
1 parent 5ccc910
commit ea7e72b
Show file tree

Hide file tree

Showing 41 changed files with 16,966 additions and 3,969 deletions.
diff --git a/.gitignore b/.gitignore
@@ -20,3 +20,4 @@ yarn-debug.log*
 yarn-error.log*
 
 .idea
+package-lock.json
diff --git a/...forcement-and-kafka-watcher-no-netpols.md → ...oud-with-enforcement-and-kafka-watcher.md b/...forcement-and-kafka-watcher-no-netpols.md → ...oud-with-enforcement-and-kafka-watcher.md
@@ -4,7 +4,7 @@ If no Kubernetes clusters are connected to your account, click the "connect your
     1. Follow the instructions to install Otterize <b>with enforcement on</b> (not in shadow mode) for this tutorial. In other words, <b>omit</b> the following flag in the Helm command: `--set intentsOperator.operator.mode=defaultShadow` 
     2. And <b>add</b> the following flags to the Helm command: 
        ```
-       --set intentsOperator.operator.enableNetworkPolicyCreation=false \ 
+       --set intentsOperator.operator.enableNetworkPolicyCreation=false \
        --set networkMapper.kafkawatcher.enable=true \
        --set networkMapper.kafkawatcher.kafkaServers={"kafka-0.kafka"}
        ```

diff --git a/docs/_common/install-otterize-kafka.md b/docs/_common/install-otterize-kafka.md
@@ -3,7 +3,7 @@ Use Helm to install the latest version of Otterize:
    helm repo add otterize https://helm.otterize.com
    helm repo update
    helm install -n otterize-system --create-namespace \
-   --set intentsOperator.operator.enableNetworkPolicyCreation=false otterize otterize/otterize-kubernetes
+   --set intentsOperator.operator.mode=defaultShadow --set intentsOperator.operator.enableNetworkPolicyCreation=false otterize otterize/otterize-kubernetes
    ```
 
 You can add the `--wait` flag for Helm to wait for deployment to complete and all pods to be Ready, or manually watch for all pods to be `Ready` using `kubectl get pods -n otterize-system -w`.
diff --git a/docs/_common/install-otterize-no-netpols-with-kafka-watcher.md b/docs/_common/install-otterize-no-netpols-with-kafka-watcher.md
diff --git a/docs/quick-tutorials/k8s-istio-authorization-policies.mdx b/docs/quick-tutorials/k8s-istio-authorization-policies.mdx
@@ -113,15 +113,68 @@ Otterize will add an Istio authorization policy to allow the intended call
 
 ### See it in action
 
-Keep an eye on the logs being tailed in the **[client-other]** terminal window,
+<details>
+<summary>Optional: check deployment status</summary>
+Check that the client and server pods were deployed
+
+```bash
+kubectl get pods -n otterize-tutorial-istio
+```
+You should see
+```
+NAME                           READY   STATUS    RESTARTS   AGE
+client-68b775f766-749r4         2/2     Running   0          32s
+nginx-c646898-2lq7l             2/2     Running   0          32s
+other-client-74cc54f7b5-9rctd   2/2     Running   0          32s
+```
+</details>
+
+monitor both client attempts to call the server with additional terminal windows,
+so we can see the effects of our changes in real time.
+
+2. **Open a new terminal window [client]** and tail the client log:
+```bash
+kubectl logs -f --tail 1 -n otterize-tutorial-istio deploy/client
+```
+<details>
+<summary>Expected output</summary>
+
+At this point the client should be able to communicate with the server:
+```
+Calling server...
+HTTP/1.1 200 OK
+...
+hello from /client-path
+```
+</details>
+
+3. **Open another terminal window [client-other]** and tail the other-client log:
+```bash
+kubectl logs -f --tail 1 -n otterize-tutorial-istio deploy/other-client
+```
+
+<details>
+<summary>Expected output</summary>
+
+At this point the client should be able to communicate with the server:
+```
+Calling server...
+HTTP/1.1 200 OK
+...
+hello from /other-client-path
+```
+
+</details>
+
+Keep an eye on the logs being tailed in the **[other-client]** terminal window,
    and apply this `intents.yaml` file in your **main terminal window** using:
    ```shell
    kubectl apply -f https://docs.otterize.com/code-examples/istio-authorization-policies/intents.yaml
    ```
 :::tip
 Client intents are the cornerstone of [intent-based access control (IBAC)](https://otterize.com/ibac).
 :::
-2. You should quickly see in the **[client-other]** terminal that it times out when calling the server,
+2. You should quickly see in the **[other-client]** terminal that it times out when calling the server,
    as expected since it didn't declare its intents:
 ```bash
 Calling server...
@@ -130,9 +183,8 @@ HTTP/1.1 200 OK
 hello from /other-client-path  # <- before applying the intents file
 # highlight-start
 Calling server...              # <- after applying the intents file
-RBAC: access denied
-Calling server...
-RBAC: access denied
+curl timed out
+Terminated
 # highlight-end
 ```
 

diff --git a/docs/quick-tutorials/k8s-istio-watcher.mdx b/docs/quick-tutorials/k8s-istio-watcher.mdx
@@ -1,6 +1,6 @@
 ---
-sidebar_position: 5
-title: Istio traffic mapping
+sidebar_position: 6
+title: Istio HTTP-level access mapping
 ---
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
@@ -83,7 +83,7 @@ kubectl label namespace otterize-tutorial-istio-mapping istio-injection=enabled
 Apply this configuration in the `istio-system` namespace, propagating it to all namespaces covered by the mesh.
 
 ```
-kubectl apply -f https://docs.otterize.com/code-examples/network-mapper/istio-telemetry-enablement.yaml
+kubectl apply -f https://docs.otterize.com/code-examples/network-mapper/istio-telemetry-enablement.yaml -n istio-system
 ```
 
 ```yaml

diff --git a/docs/quick-tutorials/k8s-kafka-mapping.mdx b/docs/quick-tutorials/k8s-kafka-mapping.mdx
@@ -0,0 +1,156 @@
+---
+sidebar_position: 4
+title: Kafka topic-level access mapping
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import styles from '/src/css/styles.module.css';
+
+With its Kafka watcher enabled, the network mapper allows you to map topic-level access to Kafka servers within your Kubernetes cluster.
+This provides a clear picture of which Kafka topics are being accessed and with which operations.
+In this tutorial, we will:
+
+- Deploy a Kafka broker, and three clients that call it.
+- Discover which topics are being accessed by those clients, and with which operations, using the Otterize network mapper's Kafka watcher.
+
+We will **not** be doing any access control in this demo, just purely mapping client-to-Kafka access at the topic and operation level.
+
+## Prerequisites
+
+<details>
+<summary>Prepare a Kubernetes cluster</summary>
+
+Before you start, you'll need a Kubernetes cluster. Having a cluster with a [CNI](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) that supports [NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) isn't required for this tutorial, but is recommended so that your cluster works with other tutorials.
+
+{@include: ../_common/cluster-setup.md}
+</details>
+
+You can now install Otterize in your cluster, and optionally connect to Otterize Cloud. Connecting to Cloud lets you
+see what's happening visually in your browser, through the "access graph".
+
+So either forego browser visualization and:
+
+<details>
+<summary>Install Otterize in your cluster with the Kafka watcher component enabled, <b>without</b> Otterize Cloud</summary>
+
+```
+helm repo add otterize https://helm.otterize.com
+helm repo update
+helm install otterize otterize/network-mapper -n otterize-system --create-namespace \
+--set kafkawatcher.enable=true \
+--set kafkawatcher.kafkaServers={"kafka-0.kafka"}
+```
+
+</details>
+
+Or choose to include browser visualization and:
+
+<details>
+<summary>Install Otterize in your cluster, <b>with</b> Otterize Cloud</summary>
+
+####  Create an Otterize Cloud account
+
+{@include: ../_common/create-account.md}
+
+#### Install Otterize OSS, connected to Otterize Cloud
+
+{@include: ../_common/install-otterize-from-cloud-with-enforcement-and-kafka-watcher.md}
+
+</details>
+
+Finally, you'll need to install the Otterize CLI (if you haven't already) to interact with the network mapper:
+
+<details>
+<summary>Install the Otterize CLI</summary>
+
+{@include: ../_common/install-otterize-cli.md}
+
+</details>
+
+## Install Kafka
+
+We will deploy a Kafka broker using Bitnami's [Helm chart](https://github.com/bitnami/charts/tree/master/bitnami/kafka).
+In the chart we will configure Kafka to:
+- Recognize the Otterize intents operator as a super user so it can configure ACLs.
+- Turn on Kafka debug logging to allow the Kafka watcher to feed topic-level client access information to the network mapper.
+
+<details>
+<summary>Expand to see the Helm values.yaml used with the Bitnami chart</summary>
+
+```yaml
+{@include: ../../static/code-examples/kafka-mapping/helm/values.yaml}
+```
+</details>
+
+The following command will deploy a Kafka broker with this chart:
+ ```bash
+ helm repo add bitnami https://charts.bitnami.com/bitnami
+ helm repo update
+ helm install --create-namespace -n kafka \
+   -f https://docs.otterize.com/code-examples/kafka-mapping/helm/values.yaml kafka bitnami/kafka --version 21.4.4
+ ```
+
+## Deploy demo to simulate traffic
+
+Let's add a few services that will access our Kafka server, and see how the network mapper builds the access map:
+- One service named "**client**".
+- One service named "**client-2**".
+
+To deploy these services, use:
+```shell
+kubectl apply -n otterize-tutorial-kafka-mapping -f https://docs.otterize.com/code-examples/kafka-mapping/all.yaml
+```
+
+Each of these services is built to periodically call the Kafka broker we deployed. Because that broker has the Otterize OSS Kafka watcher enabled and feeding data to the network mapper, we can query the network mapper directly to see the map it has built up.
+
+```shell
+otterize network-mapper list -n otterize-tutorial-kafka-mapping
+```
+
+We expect to see:
+- `client` consuming from `mytopic`.
+- `client-2` producing to `mytopic`.
+
+And indeed:
+```shell
+client in namespace otterize-tutorial-kafka-mapping calls:
+  - kafka in namespace kafka
+    - Kafka topic: transactions, operations: [describe]
+    - Kafka topic: mytopic, operations: [describe consume]
+client-2 in namespace otterize-tutorial-kafka-mapping calls:
+  - kafka in namespace kafka
+    - Kafka topic: transactions, operations: [describe]
+    - Kafka topic: mytopic, operations: [produce describe]
+```
+
+If you've attached Otterize OSS to Otterize Cloud, go back to see the [access graph in your browser](https://app.otterize.com).
+**To only see Kafka information**, make sure to de-select the 'Use in access graph' settings for network policies and Istio policies, and leave Kafka ACLs selected, like so:
+![Access graph settings](/img/quick-tutorials/kafka-mapping/settings.png)
+
+![Access graph](/img/quick-tutorials/kafka-mapping/discovered.png)
+Only the arrows between the clients and the Kafka are green, because we've selected Kafka ACLs for access graph. The other arrows were detected through network mapping, but since there's no Kafka mapping for those arrows, they are grayed out.
+
+Clicking on a specific arrow between a client and the broker reveals which topic and operations are being accessed.
+
+## What did we accomplish?
+Enabling the Kafka watcher component of the network mapper shows which clients connect to running Kafka servers, the topics they access, and the operations they undertake on those topics.
+
+You can consume this information in various ways:
+* Visually via the access graph, where shadow mode shows you what would in enforcement mode before actually turning on enforcement, and auto-generating client intents to bootstrap rolling out IBAC.
+* [Via the CLI](/reference/cli): from the network mapper directly or the cloud.
+* [Via the API](https://app.otterize.com/api/rest/v1beta).
+
+
+## What's next
+- Try our [secure access for Kafka](/quick-tutorials/k8s-kafka-mtls) tutorial
+- Follow [a more visual tutorial](/quick-visual-tutorials/visual-ibac-kafka-k8s) for securing Kafka with IBAC in a demo ecommerce application.
+- Learn how to easily secure pod-to-pod access with IBAC using Kubernetes network policies, in [a hands-on tutorial](/quick-tutorials/k8s-network-policies) or [a more visual tutorial](/quick-visual-tutorials/visual-ibac-network-policies).
+
+## Teardown
+
+To remove the deployed examples run:
+```bash
+helm uninstall otterize -n otterize-system
+helm uninstall kafka -n kafka
+helm delete ns otterize-tutorial-kafka-mapping
+```