KBrowse: initial release

This commit is intended to be the initial open source release of KBrowse.

Test Plan: run procedures described in the README.

.eslintrc

@@ -0,0 +1,5 @@
+{
+    "env": {
+        "es6": true
+    }
+}

.gitignore

@@ -0,0 +1,34 @@
+#
+# Copyright 2017-present Pandora Media, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+/.lein-*
+/.nrepl-port
+/.venv*
+/classes
+/checkouts
+/dist
+/kafka
+/kafka-logs
+/node_modules
+/target
+/tmp-tests
+/var
+/zookeeper
+*.class
+*.jar
+*.log
+*.swp
+*.tgz

CONTRIBUTING.txt

@@ -0,0 +1,51 @@
+Contributing to Pandora Media Open Source Projects
+=========================================
+
+Pandora welcomes contributions to our [open source projects on Github](https://github.com/pandoramedia/).
+
+Issues
+------
+
+Feel free to submit issues and enhancement requests.
+
+Contributing
+------------
+
+Please refer to each project's style guidelines and guidelines for submitting patches and additions (if applicable). In general, we follow the "fork-and-pull" Git workflow.
+
+ 1. **Fork** the repo on GitHub
+ 2. **Clone** your fork of the project to your own machine
+ 3. **Commit** changes to your own branch
+ 4. **Push** your work back up to your fork
+ 5. Submit a **Pull request** so that we can review your changes (please include a title and a brief description)
+
+NOTE: Be sure to merge the latest from "upstream" before making a pull request!
+
+Copyright and Licensing
+-----------------------
+
+Pandora open source projects are typically licensed under the Apache 2.0 license.
+
+Pandora does not require you to assign the copyright of your contributions, you retain the copyright. Pandora does require that you make your contributions available under the Apache license in order to be included in the main repo.
+
+If appropriate, include the Apache 2.0 license summary at the top of each file along with the copyright info. If you are adding a new file that you wrote, include your name in the copyright notice in the license summary at the top of the file.
+
+## License Summary
+
+You can copy and paste the Apache 2.0 license summary from below.
+
+```
+Copyright 2017-present by Pandora Media, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+```

LICENSE → LICENSE.txt

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright 2017-present Pandora Media, Inc.
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

NOTICE.txt

@@ -0,0 +1,7 @@
+This project contains the script 'lein' for the optional dependency 'Leiningen', 
+a project for automating Clojure projects, which can be obtained at:
+
+  * LICENSE:
+    * https://github.com/technomancy/leiningen/blob/master/COPYING (Eclipse Public License v1.0)
+  * HOMEPAGE:
+    * https://github.com/technomancy/leiningen

README.md

@@ -0,0 +1,70 @@
+# KBrowse
+This is a tool for exploring data on an Apache Kafka cluster.
+
+![Screenshot](screenshot.png)
+
+# Why
+Have you ever written a loop like this, for debugging purposes?
+```
+for record in topic:
+    if record.key == "foo":
+        print record
+```
+
+KBrowse provides this as a service, along with a web console.
+
+Cached results can be shared by copy/pasting the URL.
+
+# Usage
+Defaulting to localhost:
+```
+# Local data
+./run-zookeeper-and-kafka
+
+# Server
+./lein run server
+
+# Web Console
+open http://localhost:4000
+```
+
+Using environment overrides:
+```
+KAFKA_TIMEOUT=30000 KAFKA_BOOTSTRAP_SERVERS='foo=foo.localhost:9092' ./lein run server
+```
+
+Different yaml config:
+```
+CONFIG=config/default-with-avro.yml ./lein run server
+```
+
+Uberjar:
+```
+# Build
+./lein uberjar
+
+# Run
+CONFIG=foo.yml java $JVM_OPTS -jar kbrowse-*-SNAPSHOT-standalone.jar server
+```
+
+# Tests
+```
+./run-tests
+```
+
+# Deserialization
+KBrowse deserialization options include String, Msgpack, and Avro (with schemas looked up from the KAFKA_SCHEMA_REGISTRY_URLS).
+
+An attempt is made to parse the deserialized results into JSON, before streaming back as part of the response.
+
+# Stateful URL
+The web console's state is stored as string-serialized JSON in the browser's window location.  This is an effort to distinguish it visually from what might traditionally be a web form query.  When the web console loads, the query state is initialized from URL state, and an initial query is sent to the server.  If there are any cached results, they are displayed immediately.
+
+This design is intended to support copy/paste-able URLs, so folks can share their findings, with the context baked in.  This can save duplicated query time, and also allows verification of the query itself.
+
+Some browsers may have a limit on the window location length, which could lead to a corrupt state.  This trade-off is accepted, to allow for share-able contexts.
+
+# Browser Support
+The web console uses Mozilla's Fetch API, and ES6 keywords such as `let` and `const`.
+
+It is believed that this limits support to newer versions of Chrome, Firefox, and Safari.

config/default-with-avro.yml

@@ -0,0 +1,28 @@
+#
+# Copyright 2017-present Pandora Media, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+port: 4000
+kafka-topics-cache-sleep-seconds: 300
+kafka-timeout: 5000
+kafka-seek-to-timestamp-tolerance: 10000
+cache-max-items: 100
+cache-ttl-minutes: 60
+cache-item-size-limit: 4194304 # 4 MB
+stop-consumers-after-n-seconds: 86400
+kafka-bootstrap-servers:
+    "local": "localhost:9092"
+kafka-schema-registry-urls:
+    "local": "http://localhost:8081"

config/default.yml

@@ -0,0 +1,26 @@
+#
+# Copyright 2017-present Pandora Media, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+port: 4000
+kafka-topics-cache-sleep-seconds: 300
+kafka-timeout: 5000
+kafka-seek-to-timestamp-tolerance: 10000
+cache-max-items: 100
+cache-ttl-minutes: 60
+cache-item-size-limit: 4194304 # 4 MB
+stop-consumers-after-n-seconds: 86400
+kafka-bootstrap-servers:
+    "local": "localhost:9092"

fetch-kafka-tgz

@@ -0,0 +1,56 @@
+#!/bin/bash
+
+#
+# Copyright 2017-present Pandora Media, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Fetch the Kafka binaries into a local "kafka" dir.
+
+set -e -o pipefail
+
+function verify_md5_hash {
+    FILE=$1
+    HASH=$2
+    set +e
+    if type md5 > /dev/null; then
+        # OSX
+        set -e
+        if [ "$(md5 $FILE | awk '{ print $4 }')" == "$HASH" ]; then
+            return 0
+        else
+            echo md5 hash checked failed on $FILE
+            return 1
+        fi
+    elif type md5sum > /dev/null; then
+        # Linux
+        set -e
+        if [ "$(md5sum $FILE  | awk '{ print $1 }')" == "$HASH" ]; then
+            return 0
+        else
+            echo md5sum hash checked failed on $FILE
+            return 1
+        fi
+    else
+        echo verify_md5_hash_failed: could not find md5 or md5sum
+        return 1
+    fi
+}
+
+TGZ=kafka_2.11-1.1.0.tgz
+if [ ! -f $TGZ ]; then
+    echo Downloading Kafka...
+    curl http://www-us.apache.org/dist/kafka/1.1.0/$TGZ -o $TGZ
+    verify_md5_hash $TGZ 9487884ea6908e4dd15fa969ffcc2f9c
+fi