Skip to content
This repository has been archived by the owner on Mar 24, 2021. It is now read-only.

Releases: Parsely/pykafka

2.3.0

22 Mar 18:06
Compare
Choose a tag to compare

Minor Version Features

  • Added the ManagedBalancedConsumer class, which performs balanced consumption
    using the Kafka 0.9 Group Membership API
  • Added the managed keyword argument to Topic.get_balanced_consumer to access
    ManagedBalancedConsumer
  • Added a compacted_topic kwarg to BalancedConsumer to make it smarter about
    offset ordering for compacted topics
  • Added methods to Broker that use the Group Membership API
  • Changed the terminology "offset manager" to "group coordinator" to match updated
    Kafka jargon
  • Added new exception types from Kafka 0.9
  • Added auto_start keyword argument to Producer to match the consumer interface
  • Added max_request_size keyword argument to Producer to catch large messages
    before they're sent to Kafka
  • Added protocol functions for the Group Membership API
  • New SimpleConsumer keyword arguments: compacted_topic, generation_id,
    consumer_id

Bug Fixes

  • Fixed a bug in Travis config causing tests not to run against Kafka 0.9
  • Upgraded to non-beta gevent version
  • Allowed a single Broker instance to maintain multiple connections to a broker
    (useful when multiple consumers are sharing the same KafkaClient)
  • Allowed switchable socket implementations when using gevent
  • Handled TypeError during worker thread shutdown to avoid nuisance messages
  • Limited Producer.min_queued_messages to 1 when sync=True
  • Monkeypatched a bug in py.test causing tests to be erroneously skipped

Miscellaneous

  • Added an issue template

2.2.1

19 Feb 19:44
Compare
Choose a tag to compare

Bug Fixes

  • Fixed Travis issues related to PyPy testing
  • Fixed deprecated dependency on gevent.coros
  • Enabled caching in Travis for pip, librdkafka, and kafka installations
  • Fixed a crash during metadata updating related to zookeeper fallback
  • Unified connection retry logic in Cluster
  • Raised an exception if consumer offset reset fails after maximum retries
  • Fixed a bug allowing get_delivery_report to indefinitely block produce()
  • Fixed a bug causing producers to drop to_retry messages on stop()
  • Added retry logic to offset limit fetching

2.2.0

26 Jan 21:42
Compare
Choose a tag to compare

Minor Version Features

  • Added support for gevent-based concurrency in pure cpython
  • Allowed ZooKeeper hosts to be specified directly to KafkaClient instead of
    being treated as a fallback

Bug Fixes

  • Fixed a bug causing RLock-related crashes in Python 3
  • Used the more stable sha1 hash function as the default for
    hashing_partitioner
  • Fixed a bug in the meaning of linger_ms in the producer

2.1.2

08 Jan 22:05
Compare
Choose a tag to compare

Features

  • Allowed consumers to run with no partitions

Bug Fixes

  • Fixed a bug causing consumers to hold outdated partition sets
  • Handled some previously uncaught error codes in SimpleConsumer
  • Fixed an off-by-one bug in message set fetching
  • Made consume() stricter about message ordering and duplication

2.1.1

11 Dec 23:11
Compare
Choose a tag to compare

Features

  • Improved unicode-related error reporting in several components
  • Removed the ZooKeeper checker thread from the BalancedConsumer
  • Added a test consumer CLI to kafka_tools

Bug Fixes

  • Fixed a memory leak in the rdkafka-based consumer
  • Fixed offset committing to work against Kafka 0.9
  • Improved the reliability of the Kafka test harness

Miscellaneous

  • Simplified the Travis test matrix to handle testing against multiple Kafka versions

2.1.0

25 Nov 23:11
Compare
Choose a tag to compare

Features

  • Addded an optional C extension making use of librdkafka for enhanced producer and
    consumer performance
  • Added a delivery report queue to the Producer allowing per-message errors
    to be handled
  • Added a callback indicating that the BalancedConsumer is in the process of rebalancing

Bug Fixes

  • Fixed a longstanding issue causing certain tests to hang on Travis
  • Fixed a bug causing the default error handles in the consumer to mask unknown error
    codes
  • Moved the Message class to using __slots__ to minimize its memory footprint

2.0.4

23 Nov 20:29
Compare
Choose a tag to compare

Features

  • Allowed discovery of Kafka brokers via a ZooKeeper connect string supplied to
    KafkaClient

Bug Fixes

  • Made BalancedConsumer's ZooKeeper watches close quietly on consumer exit
  • Disconnect sockets in response to any socket-level errors
  • Fixed HashingPartitioner for python 3

2.0.3

10 Nov 23:54
Compare
Choose a tag to compare

Features

  • Raise exceptions from worker threads to the main thread in BalancedConsumer
  • Call stop() when BalancedConsumer is finalized to minimize zombie threads

Bug Fixes

  • Use weak references in BalancedConsumer workers to avoid zombie threads creating
    memory leaks
  • Stabilize BalancedConsumer.start()
  • Fix a bug in TopicDict.values() causing topics to be listed as None
  • Handle IOError in BrokerConnection and socket.recvall_into
  • Unconditionally update partitions' leaders after metadata requests
  • Fix thread-related memory leaks in Producer
  • Handle connection errors during offset commits
  • Fix an interpreter error in SimpleConsumer

2.0.2

29 Oct 18:07
Compare
Choose a tag to compare

Features

  • Switched the BalancedConsumer to using ZooKeeper as the single source of truth
    about which partitions are held
  • Made BalancedConsumer resilient to ZooKeeper failure
  • Made the consumer resilient to broker failure

Bug Fixes

  • Fixed a bug in BrokerConnection causing the message length field to
    occasionally be corrupted
  • Fixed a bug causing RequestHandler worker threads to sometimes abort
    before the request was completed
  • Fixed a bug causing SimpleConsumer to hang when the number of brokers in
    the cluster goes below the replication factor

2.0.1

19 Oct 23:49
Compare
Choose a tag to compare

Features

  • Added support for python 3.5
  • Added iteration to the BalancedConsumer
  • Disallowed min_queued_messages<1 in Producer
  • Made SimpleConsumer commit offsets on stop()
  • Supported None in message values

Bug Fixes

  • Improved BalancedConsumer's handling of an external KazooClient instance
  • Fixed kafka_tools.py for Python 3
  • Removed the unused use_greenlets kwarg from KafkaClient
  • Improved Cluster's ability to reconnect to brokers during metadata updates
  • Fixed an interpreter error in conncection.py
  • Fixed failure case in Producer when required_acks==0
  • Fixed a bug causing SimpleConsumer to leave zombie threads after disconnected brokers
  • Improved SimpleConsumer's worker thread exception reporting
  • Simplified SimpleConsumer's partition locking logic during fetch by using RLock
  • Fixed SimpleConsumer off-by-one error causing lag to never reach 0

Miscellaneous

  • Switched from Coveralls to Codecov for converage tracking