Unable to request new process from fork server

[vanhauser-thc]

I get the error "Unable to request new process from fork server" when running the cxxfilt example.

dmesg shows that pt-proxy-fast is crashing:

[  113.654569] Cannot allocate proxy vma
[  113.654584] pt-proxy-fast[2194]: segfault at 1 ip 00007ffff787a183 sp 00007fffffffe1a8 error 4 in libc-2.19.so[7ffff77f3000+1be000]

I am running 14.04.05 with kernel 4.4.0-148-generic (native, no docker or VM).
ldd on cxxfilt shows the right ld.so in the afl-.../pt_mode/glibc/build directory and it exists.

full dmesg output:

[   49.997764] ptmodule: loading out-of-tree module taints kernel.
[   49.997803] ptmodule: module verification failed: signature and/or required key missing - tainting kernel
[   49.998075] The PT supports 36 ToPA entries and 2 address ranges for filtering
[  109.626931] Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.
[  113.340476] Unsafe core_pattern used with suid_dumpable=2. Pipe handler or fully qualified core dump path required.
[  113.554696] Proxy start with PID 2194
[  113.582989] Target confirmed: ./test_progs/binutils-2.29/build/binutils/cxxfilt, ptm ffff88021f634000
[  113.645876] Fork server path ./test_progs/binutils-2.29/build/binutils/cxxfilt and pid 2195
[  113.645878] The CPU ID for fork server is 0
[  113.654569] Cannot allocate proxy vma
[  113.654584] pt-proxy-fast[2194]: segfault at 1 ip 00007ffff787a183 sp 00007fffffffe1a8 error 4 in libc-2.19.so[7ffff77f3000+1be000]
[  113.654619] Exit of the proxy process
[  113.654619] In total 0 runs
[  113.654620] Release trace point

[vanhauser-thc]

I found the issue.
do not supply any kernel options, e.g. for meltdown or spectre bypass for more performance

then it works, on the stock 4.4.0-31 kernel as well as on the "latest" 4.4.0-148

[vanhauser-thc]

also kernel.random_va_space may not be 0 then the OOM happens too

[vanhauser-thc]

and a final hint: the stock -31 kernel is faster for fuzzing than the -148 one ... for whatever reason.