gearmand hanging and non responsive after OMD 3.30 upgrade

[infraweavers]

After upgrading to OMD 3.30 from 3.10 we find that gearmand hangs and is no longer able to be connected to, meaning all the checks timeout. We use mod_gearman to provide dupserver which we then submit the results to our secondary OMD box.

omd restart gearmand fails to kill the process and the only solution is to kill -9 gearmand.

We have core files of gearmand when it's in this state and ran strace against it, which showed no activity occurring in the process.

The error from running gearman_top when its hung is:

failed to connect to address localhost and port 4730: Interrupted system call.

which is very unusual; if gearmand is stopped or unavailable, connection refused is the error.

We think this probably related to the upgrade from gearmand 0.33 that we think we occurred in OMD 3.20? Where is the best place to discuss the issue, is here fine?

[sni]

the place to discuss this is fine. I see this on one box occasionally as well but haven't found time to dig deeper into it.

[infraweavers]

We've tried to reproduce this in a lab by copying the naemon check configuration and all the plugins onto a test OMD 3.30 install, the problem has not reocurred in the lab; so we're now wondering if it's somethign specific with the output of one of our checks or similiar. So we've built another OMD 3.30, this time in the same environment as the ones we upgraded previous and it's running all the same checks in parallel to OMD3.10; the only difference in configuration between the 2 is Keepalived and Postfix are both disabled on the OMD3.30; and the dupserver is commented out on OMD3.30 at the moment. If we can't reproduce the problem in a few hours, we'll create another OMD3.30 and configure the dupserver so it is identical to the upgraded pair we had the issue on above.

[infraweavers]

This has just been reproduced now, so presumably the specific checks we're running are important.

[sni]

thats strange, because mod-gearman encrypts and converts the crypted payload with base64 to avoid such problems. Except for the uniq identifier. So it might have to do something with a specific host or servicename.

[infraweavers]

We've also just had this scenario (-1 jobs) appear whilst trying to narrow it down:

Looks like gearadmin --status wraps the integer:

[infraweavers]

@sni we use gcore to get the coredumps, we think we need a debug build of gearmand with symbols to feed into gdb do you know how we can build one or get the required files?

[infraweavers]

We think we got this working, looks like just building gearmand from OMD 3.30's release commit was enough:

[sni]

yeah, you can simply run make from the packages/gearmand folder. Eventually you need to run ./configure from the base folder to install all build dependencies. And you might want to adjust some cflags to increase debug symbols if necessary.

Seems that gearmands internal state gets confused somehow.

[infraweavers]

We just got it reproduced again, however this time we were tailing the gearmand log,

We saw the gearman_top have -1 in the check_results, that correllated with the log appearing; shortly after the gearmand stopped responding completely.

[infraweavers]

We managed to catch and attach to this happening, and it's basically in an infinite loop between:
https://github.com/gearman/gearmand/blob/master/libgearman-server/worker.cc#L124-L131

[sni]

great debugging, i guess its time to get the gearmand people involved.

[infraweavers]

We've opened an issue here: gearman/gearmand#301 we think that's the right place for it

[infraweavers]

@sni We do use pnp_gearman_worker for our graphing on the affected environment; we'll make this change (dd8b990) on our test OMD 3.30 box and see if it makes any difference to it.

[sni]

are you using pnp in gearman mode?

[infraweavers]

Yes

[sni]

ok, then it might be the same issue.

[infraweavers]

@sni We patched our test OMD 3.30 box on the 28th and have it running since. It doesn't seem to have locked up with interrupted system call in the few days we've been running it, whereas before it would almost certainly have done it once by now. So it seems like dd8b990 may have worked around the gearmand bug in our environment. We'll keep testing it for a bit and see if anything changes.

[sni]

same here, no outage since then. Looks good.

[infraweavers]

@sni just tried promoting one of our actual production boxes up to 3.30 and applied dd8b990, unfortunately it just broke again with the exact same behaviour as before, so it doesn't look like it's fixed it for us :(

[sni]

still no issues here since with the latest nightly

[infraweavers]

Interesting, we're testing with the nightly rather than manually applying dd8b990 to see if we get it again

[infraweavers]

@sni Just checking through the mod_gearman code for anything looking "weird"; noticed this: https://github.com/sni/mod_gearman/blob/master/common/gearman_utils.c#L97. Is that definitely the connection to gearmand network timeout and not a job timeout (i.e. CAN_DO_TIMEOUT in gearman parlance) ?

[infraweavers]

Nightly has been running for a few hours now and we've not seen the problem yet

[infraweavers]

We're going to upgrade our secondary at our first site to the nightly and see if the stability remains. It's looking pretty good so far, however!

[infraweavers]

Looking stable so far, there does seem to be a pause in the behaviour of checks when the dupserver is unavailable; however that is a totally separate issue from this. The good news is, it does seem to be fixed so far...

[infraweavers]

Just reporting in, this has been stable for another week; I'm pretty convinced this has worked around the gearmand problem now :)

[sni]

Thats great to hear. Will close this then