advice needed: producer under gunicorn never sends messages

[twillis]

TLDR: is there a recommended way to use pykafka in a web app to produce messages?

I need to produce some messages as part of a web application and I can do so by creating the connection when the request comes in, but the overhead is too much with sync=true. messages never arrive when connection/topic/producer are module level.

The app in question is running in gunicorn with preload_app=True and worker_class="gthread"

Right now I'm getting around this by spawning a thread and producing the message for the request there but I have concerns that spawning an additional thread per request /response to write message would cause issues under a production load.

I would be running into the same kinds of problems with the database if I was spawning a connection per thread or even worse, per request, however the author(s) of sqlalchemy gives me a connection pool and it works like a charm even with 48 worker threads per 8 processes.

I'm thinking something similar is needed for kafka but i'm kind of surprised it hasn't been written already.

This leads me to believe that maybe i'm on the wrong track here and there is a way in pykafka for producing messages in the context of a web app without the web developer having to know all the intricacies of threading/processes with this library.

Using something like celery seems redundant here and I would rather not take on the complexity of putting rabbitmq or redis in our infrastructure so I can produce messages into kafka.

[twillis]

here's source that demonstrates the problem. https://github.com/twillis/kafka_web