v0.4.1 - Stability improvements
Lots of stability improvements in this release. Here are the changes:
Functional changes:
- Graphs could sometimes be difficult to read because you would see non-whole numbers on the y-axis legend, when facts being drawn are always whole numbers. Set y-axis to draw only whole numbers in the legend to make the graph easier to interpret.
- Respect command-line arg kafkaOffsetForceFromStart, starting consumer offset listener clients from the beginning of the log by implementin a ConsumerRebalanceListener.
Stability improvements:
- Created function tryParseOffsetMessage to attempt to parse a kafka offset message retrieved from the internal committed offset topic:
- Handles messages of other types and questionable correctness.
- Added 100% unit-test coverage for this new function.
- Add robustness to the log-end-offset getter thread:
- No longer shutting down the application on error. Instead, closing and destroying the client and re-creating it.
- Sleeping on error before re-creating client and continuing to process
- Deal with thread-safety issues on shared memory between threads that retrieve data from Kafka.
- Stopped polluting consumer groups in zookeeper by not creating a unique consumer group name for the consumer-offset and log-end-offset listener at each client instantiation.
- Improved createNewAdminClient code, simplifying the error paths and property calling close on error.
- Re-factored some of the error handling paths, simplifying them.
- Closing all kafka clients on error so connections do not leak.
General improvements:
- Begin to reduce usage of Zookeeper when using offsetStorage = kafka:
- Override getTopics() in KafkaOffsetGetter to retrieve topics directly from the Kafka broker
- Override getClusterVis() in KafkaOffsetGetter to retrieve cluster information directly from the Kafka broker.
- Use constants for all property in createNewKafkaConsumer().
- Fixed some bad grammar in error messages.
- Fixed silly com.twitter.util-core dependency in build.sbt.