Skip to content

troubleshooting.md

Latest commit

 

History

History
219 lines (157 loc) · 7.88 KB

troubleshooting.md

File metadata and controls

219 lines (157 loc) · 7.88 KB
Raw

Troubleshooting

Start every investigation the same way:

TOFFLOAD_STATS=stderr:1 LD_PRELOAD=/path/to/libtransparent.so ./your-server

One counter line per second tells you which of the situations below you are in.

The runtime does not seem to load at all

No [toffload] runtime 0.1.0 active banner.

Wrong path. LD_PRELOAD needs a path the loader can resolve, and it fails silently. Use an absolute path and confirm:

LD_DEBUG=libs LD_PRELOAD=$PWD/build/lib/libtransparent.so ./your-server 2>&1 | grep transparent

Setuid or setgid binary. The loader ignores LD_PRELOAD for privileged binaries. Nothing to do but run an unprivileged copy.

TOFFLOAD_QUIET is set. Then there is no banner by design; check the stats line instead.

Statically linked application. There are no dynamic symbols to interpose. The runtime cannot help a static binary.

fibers_created=0 — loaded but never engaged

The most common outcome, and usually not a bug. Fiberization starts only after the first successful accept(), and only for threads created after it.

  • Event-loop server (nginx, redis, memcached). There is no per-connection thread. Expected, safe, and no overlap is available transparently. See is-my-server-supported.md.
  • Thread pool created at startup. The workers predate the first accept(), so they stay real threads. Try TOFFLOAD_POOL=1, and test carefully: it collapses the pool onto the carrier and changes the server's concurrency model.
  • Process-per-connection (fork() per connection, like PostgreSQL). There is no pthread_create to interpose.
  • The server never accepts on a socket you expected — check it is really serving.

fibers_created climbs but offloads stays 0

Fibers are running; the offload is not being intercepted. The runtime resolves accel_run, then falls back to accel_encrypt.

nm -D --defined-only /path/to/your-backend.so | grep accel_

You should see accel_submit, accel_done, accel_release and one of accel_run / accel_encrypt.

  • C++ backend without extern "C". The names get mangled and nothing resolves. Every exported symbol needs C linkage.
  • Preload order. The backend must come before the runtime: LD_PRELOAD="./libaccel_mine.so ./libtransparent.so".
  • Statically linked backend. If the offload is compiled into the application rather than a shared object, there is no dynamic symbol to interpose. Build the backend as a .so.

io_yields stays 0, or TOFFLOAD_ENFORCE serializes nothing

Symptom: the runtime is clearly loaded and fibers_created and offloads climb, but io_yields is 0 and, with TOFFLOAD_ENFORCE=1, serialized is also 0 and unlocked shared state still loses updates.

The application's socket reads are not reaching the runtime. A distribution that compiles with _FORTIFY_SOURCE=2 or higher under gcc 12 or newer rewrites read(fd, buf, n) into __read_chk(fd, buf, n, size) whenever the destination size can be determined, and glibc exports that under its own name. Ubuntu 24.04 does this by default. The same applies to recv, recvfrom and poll.

The runtime interposes all four _chk forms, so an up-to-date build handles this. Confirm yours does:

nm -D --defined-only build/lib/libtransparent.so | grep _chk

You should see __read_chk, __recv_chk, __recvfrom_chk and __poll_chk. If they are absent the build predates the fix. As a stopgap you can rebuild the application with -U_FORTIFY_SOURCE, but upgrading the runtime is the real answer. Note that the write direction has no _chk form: glibc fortifies only calls that write into a caller's buffer.

Throughput got worse

Compare on an equal core budget. This is the usual answer. A stock thread-per-connection server on a 32-core machine spreads 64 OS threads over 32 cores; the runtime multiplexes them onto one carrier. Unpinned, you are measuring the machine, not the mechanism:

taskset -c 4,6 ./your-server                                    # stock
taskset -c 4,6 env LD_PRELOAD=... ./your-server                  # runtime

Your offload may be too short. Below roughly 50 µs on ordinary hardware, parking and resuming a fiber costs about as much as the wait it avoids. Our own demo shows 0.97× at 20 µs. Check with ACCEL_LAT_US if you are using the emulated backend.

The accelerator may be the bottleneck. Overlap helps when the CPU would stall, not when the device is saturated. With ACCEL_OP=zip on the single-threaded emulated device, throughput is ~1.05× because the device itself is the limit. Look at device utilization before blaming the runtime.

serialized is climbing. Enforce mode is doing its job, which costs throughput by design. If det_conflicts is 0, you do not need enforcement — turn it off.

It hangs

Check serialized and fibers_live. If fibers_live dropped and progress stopped, something is holding a lock it never released. That exact bug shipped once (the handler lock at EOF) and is fixed; a recurrence is a bug report.

Sub-libc blocking. A fiber that enters a raw futex or io_uring_enter blocks the carrier and everything on it. Confirm with:

strace -f -c -e trace=futex,io_uring_enter -p $(pgrep your-server)

Heavy futex traffic means category 3 — the runtime cannot yield through it. See is-my-server-supported.md.

TOFFLOAD_RT=1 is set. The realtime carrier can starve a real OS thread that a fiber is waiting on, giving priority-inversion deadlock. Do not use it outside microbenchmarks.

TOFFLOAD_REALMUTEX=1 is set. That selects the naive mutex, which deadlocks the carrier on purpose to demonstrate why the fiber-aware one exists.

Bisect with the diagnostic knobs:

TOFFLOAD_PASSIVE=1 ...   # fibers on, every other interposer off
TOFFLOAD_NOFIBER=1 ...   # everything off; if this fixes it, it is fiberization
TOFFLOAD_NOMUTEX=1 ...   # narrow to the mutex
TOFFLOAD_NOCOND=1 ...    # narrow to condition variables
TOFFLOAD_NOIO=1 ...      # narrow to socket I/O

It crashes

At startup, in a database. Almost certainly versioned-symbol interposition. Interposing pthread_cond_* without matching glibc's GLIBC_2.3.2 versions crashes MariaDB at startup — measured on 1 of 5 stock binaries. The runtime handles this with src/transparent.map and dlvsym. If you built without the version script, that is the cause. Confirm:

objdump -T build/lib/libtransparent.so | grep pthread_cond_wait
# must show GLIBC_2.3.2

Wrong errno or OpenSSL errors. If per-fiber save/restore were disabled (TOFFLOAD_NOTLS=1) this is expected. Otherwise it is a bug — please report it.

Stack overflow in a handler. Fibers get 1 MB stacks, less than the 8 MB default thread stack. A handler with deep recursion or a large stack array can overflow. This shows up as a crash inside the handler with a plausible-looking but wrong backtrace.

Results are wrong (bad > 0)

A correctness bug, not a tuning issue. Please open an issue.

Before you do, check whether the runtime is even implicated:

TOFFLOAD_NOFIBER=1 LD_PRELOAD=... ./your-server    # runtime loaded, fibers off

If bad > 0 persists with fibers off, the problem is in your application or backend, not the scheduler. If it disappears, it is ours — include the stats line, your knobs, and the server's concurrency model.

One likely application-side cause: unlocked shared state read-modify-written across the offload. That is a real hazard the runtime creates; see safety.md. Run with the detector and look at det_conflicts before concluding the runtime is broken.

Getting help

Open an issue with:

  • the exact command line, including every TOFFLOAD_* and ACCEL_* variable
  • TOFFLOAD_STATS=stderr:1 output
  • distribution, kernel, glibc, compiler
  • whether TOFFLOAD_NOFIBER=1 changes the behaviour