Confluent's Kafka client for Python wraps the librdkafka C library, providing full Kafka protocol support with great performance and reliability.
The Python bindings provides a high-level Producer and Consumer with support for the balanced consumer groups of Apache Kafka 0.9.
See the API documentation for more info.
License: Apache License v2.0
Producer:
from confluent_kafka import Producer
p = Producer({'bootstrap.servers': 'mybroker,mybroker2'})
for data in some_data_source:
p.produce('mytopic', data.encode('utf-8'))
p.flush()
High-level Consumer:
from confluent_kafka import Consumer
c = Consumer({'bootstrap.servers': 'mybroker', 'group.id': 'mygroup',
'default.topic.config': {'auto.offset.reset': 'smallest'}})
c.subscribe(['mytopic'])
while running:
msg = c.poll()
if not msg.error():
print('Received message: %s' % msg.value().decode('utf-8'))
c.close()
See examples for more examples.
- Python >= 2.7 or Python 3.x
- librdkafka >= 0.9.1
Install from PyPi:
pip install confluent-kafka
Install from source / tarball:
pip install .
python setup.by build
Run unit-tests:
py.test
NOTE: Requires py.test, install by pip install pytest
Run integration tests:
examples/integration_test.py <kafka-broker>
WARNING: These tests require an active Kafka cluster and will make use of a topic named 'test'.
Install sphinx and sphinx_rtd_theme packages and then:
make docs
or:
python setup.by build_sphinx
Documentation will be generated in docs/_build/