-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
3.10.0-rc1: imapd SIGABRTs inside printf_positional() when compiled with optimizations turned on (anything greater than -O0) #4972
Comments
Cyrus doesn't support compiling with LTO, though I don't remember why off the top of my head. But if it still crashes when compiled without LTO, that's quite interesting. I'll try to reproduce the crash myself and see what shakes out. |
The top of the call stack looks a lot like #1981, though the bottom is different. I wonder if we have a bad format string in the IMAP capabilities table. If you turn off |
This is not fully tested yet but might get you moving forward:
|
For me, with -O3, FORTIFY_SOURCE, and this patch, one Cassandane test fails in a way that I don't understand yet, and ~50 or so tests fail because of problems examining syslog. If LD_PRELOAD isn't inherited by child processes under FORTIFY_SOURCE, which seems plausible, that would explain the syslog problem. Cassandane tries to detect when its syslog replacement isn't available, but in this case it thinks it's available when it's not. I have an idea of how to make the detection more robust, though it's a distraction rather than a real problem... |
Thanks for looking into it so quickly!
Yes, there were compilation errors regarding 'attribute((always_inline, optimize("-O3")))' #4679.
Indeed, with
Also after applying the patch, it doesn't crash.
I am not sure why yet but for me the tests otherwise fail due to a missing filename in sun_path when it tries to remove a socket file, but this looks unrelated (it worked on 3.8.3 though).
It could be. If started as root, only elements without slash get read from LD_PRELOAD and searched only in standard paths. I might be hitting similar issues even on 3.8.3. I have some tests disabled where it looked related to parsing results from syslog, I may open an issue for it in the future when I get to it, otherwise, I have to patch the syslog routines so they call only fake_vsyslog() not to spam system logs on our CI test machines: https://src.fedoraproject.org/rpms/cyrus-imapd/blob/rawhide/f/patch-cassandane-no-syslog |
I think I understand the syslog problem. It's not that FORTIFY_SOURCE affects LD_PRELOAD, it's that it affects The Cassandane utils don't inherit Cyrus's build settings, so they get built without FORTIFY_SOURCE. That means that when syslog_probe runs to check whether syslog replacement is available, it calls the LD_PRELOAD'd replacement, which works, so it marks syslog replacement as available. And then the tests that look for details in syslog fail because they expect to find things that aren't there. From this discussion it looks like I can probably fix this by adding
Rather than patching code, you should be able to configure syslog on your test machines to send LOG_LOCAL6 to /dev/null, or its own file, or whatever is convenient. |
Can't just intercept If the Cassandane utils are built with FORTIFY_SOURCE then the probe fails and the syslog replacement is correctly detected as unavailable, and tests stop looking for nonexistent syslog messages. But compiling them like that is fiddly and prone to being forgotten or not synchronised with the Cyrus build. Next plan will be having Cyrus itself report when it was was built with FORTIFY_SOURCE (via cyr_buildinfo), and disabling the syslog checks in Cassandane when it was... |
from what I see __vsyslog_chk is not hidden:
so you could do the same as for the openlog/vsyslog/closelog, i.e. export __vsyslog_chk and dlsym(RTLD_NEXT, "__vsyslog_chk")
Not sure why I didn't open an issue for it when I hit the same while writing the patch to turn off syslog logging that __vsyslog_chk() got called instead of syslog() for me leading to failed tests because syslog file was 0 in size. I ended up reading the same StackOverflow thread. :D |
That's exactly what I did, and it didn't work. I spotted this in the copy of syslog.c that the SO thread linked to, which seemed to explain it, so I didn't dig further: https://github.com/lattera/glibc/blob/895ef79e04a953cac1493863bcae29ad85657ee1/misc/syslog.c#L319 It might be that with your build options, The state that I'm at now is I have a set of commits that fix the Cyrus sources so that:
I can't get a clean Cassandane run out of it though. A bunch of random tests fail each run due to Cassandane's SMTP thingy not exiting all its processes fast enough. The SMTP thingy is known to be a mess with a lot of issues, but this one is new. It's perl, so I I don't see why source fortification of Cyrus would affect it. I'm not convinced it's worth my time to chase that right now, since we already know the SMTP thingy needs a rewrite anyway, so I'll tidy up my current patches into a PR and get this moved forward. |
printf(3): > There may be no gaps in the numbers of arguments specified using '$'; > for example, if arguments 1 and 3 are specified, argument 2 must > also be specified somewhere in the format string. The bad format string caused a SIGABRT during capabilities response when Cyrus was compiled with glibc source fortification enabled. Fixes #4972
This doesn't happen with 3.8.3. Happens on Fedora 39-41. When compiled with -O0, it does not crash.
Steps to reproduce:
After building with -02, just connecting with cyradm leads to the crash:
I was trying to find out what changes would be needed for the 3.10.0 version today:
https://src.fedoraproject.org/fork/mosvald/rpms/cyrus-imapd/c/785dbc4c9f42b5385b8e43e43b33c124146d2869
and hit these crashes. All the Cassandane tests were crashing for me. Tried to turn off LTO, and compile it on f39-f41, but with the same result. I haven't managed to dig deeper yet. Maybe it rings a bell for you.
The text was updated successfully, but these errors were encountered: