Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Created function tryParseOffsetMessage to attempt to parse a kafka of…
…fset message retrieved from the internal committed offset topic: * Handles messages of other types and questionable correctness. * Added 100% unit-test coverage for this new function. Add robustness to the log-end-offset getter thread: * No longer shutting down the application on error. Instead, closing and destroying the client and re-creating it. * Sleeping on error before re-creating client and continuing to process Deal with thread-safety issues on shared memory between threads that retrieve data from Kafka. Respect command-line arg kafkaOffsetForceFromStart, starting consumer offset listener clients from the beginning of the log by implementin a ConsumerRebalanceListener. Begin to reduce usage of Zookeeper: * Override getTopics() in KafkaOffsetGetter to retrieve topics directly from the Kafka broker * Override getClusterVis() in KafkaOffsetGetter to retrieve cluster information directly from the Kafka broker. Stoped polluting consumer groups in zookeeper by not creatin a unique consumer group name for the consumer-offset and log-end-offset listener at each client instantiation. Improved createNewAdminClient code, simplifying the error paths and property calling close on error. Use constants for all property in createNewKafkaConsumer(). Fixed some bad grammar in error messages. Re-factored some of the error handling paths, simplifying them. Closing all kafka clients on error so connections do not leak. Fixed silly com.twitter.util-core dependency in build.sbt.
- Loading branch information