Trouble connecting RQ workers to Opbeat

[sburns]

Hi, thanks for Opbeat, we very much enjoy using it. We're in the process of moving our Django app to use RQ+django-rq and want to continue to get errors logged in Opbeat.

I'm trying to use opbeat.contrib.rq.register_opbeat this way...

from rq import Worker
from opbeat.contrib.django.models import client
from opbeat.contrib.rq import register_opbeat

class MyWorker(Worker):

    def work(self, burst=False):
        register_opbeat(client, self)
        super(MyWorker, self).work(burst=burst)

And I start up the worker with the custom class, ./manage.py rqworker high low medium --worker-class path.to.MyWorker

The worker is syslog'd and I've verified the handler is attached and being called...

<snip>
Feb 16 15:55:39 stage-3 ERROR 2016-02-16 09:55:39,177 worker 14785 140715382544192 AssertionError#012Traceback (most recent call last):...
Feb 16 15:55:40 stage-3 DEBUG 2016-02-16 09:55:40,221 worker 14785 140715382544192 Invoking exception handler <function send_to_opbeat at 0x7ffada5b9c08>
Feb 16 15:55:40 stage-3 DEBUG 2016-02-16 09:55:40,226 worker 14785 140715382544192 Invoking exception handler <bound method MyWorker.move_to_failed_queue of <path.to.MyWorker object at 0x7ffac2f8ed90>>
</snip>

Just for fun, I threw an assert False == True early in a view and hitting that page does send an AssertionError to opbeat, so the site is configured to send errors.

Any ideas?

[beniwohli]

@sburns sorry for the long wait. I debugged this a bit, and it seems that our async http mode isn't compatible with rq. As a workaround, you can disable the async mode like this

from opbeat.contrib.django.client import DjangoClient
from opbeat.contrib.django.models import get_client_config
from opbeat.contrib.rq import register_opbeat

from rq import Worker


class MyWorker(Worker):

    def work(self, burst=False):
        config = get_client_config()
        config['async_mode'] = False
        client = DjangoClient(**config)
        register_opbeat(client, self)
        super(MyWorker, self).work(burst=burst)

I'll leave this open for now until we find out what the root problem is

[sburns]

Thanks for the update! I'll give this a spin and let you know how it goes.

[sburns]

Hi @piquadrat this didn't work for me :(

[beniwohli]

@sburns bummer :( We'll investigate further

[Zowie]

Hi! Is there any fix for this already? Otherwise I'll just have to move to Celery :)

[sburns]

Otherwise I'll just have to move to Celery :)

We're close to that as well.

[hugorodgerbrown]

@piquadrat is this still under investigation?

[ksmandersen]

+1

[hugorodgerbrown]

Any update on this?

[hugorodgerbrown]

Any update? This is getting close to abandon territory.

[beniwohli]

Hi @hugorodgerbrown, sorry for the silence on this! Totally my fault that this fell through the cracks.

Unfortunately, we weren't able to resolve this issue, and I can't give you a timeline at the moment. However, we plan to ramp up the work on our Python agent over the coming months, and first-class support for rq is definitely a priority.

[hugorodgerbrown]

Thanks for the update @beniwohli - out of interest do you have any further detail on why / where it falls down - plenty of people on this thread available to help diagnose / fix on your behalf?

[sivabudh]

@beniwohli Please help with regards to this issue because we would love to profile our RQ worker as well.

[djm]

@sivabudh @beniwohli @ksmandersen @sburns @Zowie

I spent a little time on this and managed to get it working; it's a little bit more manual than the rest of the out of the box experience but it works just fine for us.

It involves writing your own rq exception handler and plugging it into the Worker; then contacting Opbeat with the normal client (not Django client) via sync requests. Sync requests are absolutely fine for us as we're dealing with low throughput background jobs; but be aware that the call would obviously be blocking.

Rather than inundate this post with gists: Triggering Opbeat errors from Python RQ Workers.

Hope it helps someone.