Skip to content
This repository has been archived by the owner on Mar 24, 2021. It is now read-only.

Subscribe to a regex topic pattern in pykafka? #786

Closed
vikt0rs opened this issue Apr 20, 2018 · 2 comments
Closed

Subscribe to a regex topic pattern in pykafka? #786

vikt0rs opened this issue Apr 20, 2018 · 2 comments

Comments

@vikt0rs
Copy link

vikt0rs commented Apr 20, 2018

Hello, folks,

please, suggest, if there is a possibility in pykafka to get a consumer, subscribed to the topic pattern (not a single topic) as kafka-python has ?
If not - could you please suggest a workaround for this feature?

Thanks,
Viktor

@emmettbutler
Copy link
Contributor

Consumers in pykafka can only subscribe to one topic at a time, so regex subscription from a single consumer is not supported. It's blocked by #354.

The closest you could get to this in pykafka's current form is something like this:

import re
from pykafka import KafkaClient
expr = re.compile(MY_TOPIC_REGEX)
client = KafkaClient(hosts=MY_HOSTS)
consumers = {}
for topic_name, topic in client.topics.items():
    if expr.match(topic_name):
        consumers[topic_name] = topic.get_simple_consumer()

@vikt0rs
Copy link
Author

vikt0rs commented Apr 28, 2018

@emmett9001 thanks a lot!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants