Skip to content

Commit

Permalink
Fixes PandoraMedia#18: use to-result-string in progress-message
Browse files Browse the repository at this point in the history
Avro deserialization was resulting in an exception:
```
 :cause Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
 :via
 [{:type com.fasterxml.jackson.core.JsonGenerationException
   :message Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
   :at [cheshire.generate$generate invokeStatic generate.clj 152]}]
```

This is handled in to-result-string, so we now use that instead of
invoking cheshire/generate-string directly.

----
Test Plan

As williamla points out in PandoraMedia#18, we don't currently have schema/avro testing.
That is in part because the fetch-kafka-tgz approach is retrieving
binaries that do not include Schema Registry support.

One option is to update to use a Docker approach with Schema support,
as outlined in PandoraMedia#2.  Another would be to figure out if the Confluent
downloads are intended to be used in a scripted way (atm it requires
an email, so I have only tested manually).

In the absense of proper integration tests, I have tested this change
using the Confluent binaries:
```
./bin/kafka-topics --zookeeper localhost:2181 --create --topic
kbrowse-avro --partitions 10 --replication-factor 1

echo '"k"','{"label": "v"}' | ./bin/kafka-avro-console-producer
--broker-list localhost:9092 --topic kbrowse-avro --property
parse.key=true --property key.separator=, --property
key.schema='{"type": "string", "name": "label"}' --property
value.schema='{"type": "record", "name": "value", "fields": [{"type":
"string", "name": "label"}]}'

CONFIG=config/default-with-avro.yml ./lein run server

http://localhost:4000/?{%22key%22:%22%22,%22valRegex%22:%22%22,%22bootstrapServers%22:%22localhost:9092%22,%22topic%22:%22kbrowse-avro-1%22,%22relativeOffset%22:%22%22,%22follow%22:false,%22defaultPartition%22:false,%22valueDeserializer%22:%22io.confluent.kafka.serializers.KafkaAvroDeserializer%22,%22schemaRegistryURL%22:%22http://localhost:8081%22,%22partitions%22:%22%22}
```
  • Loading branch information
Patrick Tuckey committed Jul 25, 2018
1 parent 772fd5e commit 7ed43b0
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/kbrowse/search.clj
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
(-> record
to-map
(assoc :type :offset :timestamp timestamp-date)
(cheshire/generate-string {:pretty true})
(to-result-string)
(#(str ", " %)))))

(defn maybe-send-progress
Expand Down

0 comments on commit 7ed43b0

Please sign in to comment.