[flac] Intermediate step towards OOM fuzzing

[ktmf01]

This is an intermediate step towards fuzzing of out-of-memory conditions, which will be accomplished by making the Nth malloc/calloc/realloc fail. With this intermediate step, hopefully no oss-fuzz build failures will occur

[DavidKorczynski]

Very nice! We did a similar thing for Fluent Bit fluent/fluent-bit#4689 and it helped find interesting cases! I'd be interested in knowing what results you find, as from my experience it often finds a lot of malloc mishandling.

[ktmf01]

I've already fuzzed locally, most problems were with not handling reallocs correctly. In libFLAC, on realloc failing some parts of the code expects the original memory to remain intact, other parts assume it to be freed. There were some recently added mallocs that misbehaved and a few of which failure wasn't handled. But mostly the original author did a very good job.

Anyway, I guess this kind of fuzzing might not be useful for most projects, but as libFLAC is used in embedded environments it seems a good idea to have this functioning properly.

[catenacyber]

@ktmf01 are you interested in #9902 ?

[ktmf01]

@catenacyber I'm not sure, of course FLAC already has this functionality now, I don't know what might be better with this different approach. It might be interesting for other projects to have something like this without having to alter their code.

[catenacyber]

So, how did you implement this functionality ?

[ktmf01]

FLAC uses a set of malloc, calloc and realloc derivatives that check for overflow, see here: https://github.com/xiph/flac/blob/383512d551de2ebf640a95d010b38ad58b0d12ae/include/share/alloc.h Each of these has a check embedded, only on fuzzing.

Certain fuzzers (not all) do fail checks. The fuzzer sets a counter to zero and sets a threshold. For each alloc, the counter is incremented, until the threshold is reached, and the alloc fails. See here: https://github.com/xiph/flac/blob/383512d551de2ebf640a95d010b38ad58b0d12ae/oss-fuzz/seek.cc#L73-L77 By default, these checks are disabled. So, only fuzzers that set a threshold actually use this.

Combined with various sanitizers, the code paths that are only triggered on out-of-memory conditions. So fuzzing works with ASAN, UBSAN and MSAN.

[catenacyber]

Interesting, one difference I notice is that my nallocfuzz has a pseudo random function to tell when one allocation fails (further allocations will not necessarily fail), instead of a counter and always the same threshold as you do...

Did you find interesting stuff with allocations failure and UBSAN and MSAN ?
I only found NULL dereference, memory leaks, and assertion failures so far (all with ASAN)

[ktmf01]

I don't remember to be honest. The thing is that FLAC has, in my opinion, quite a lot of places it allocates memory, and it tries to fail gracefully at every allocation. That means there are a lot of code paths that are untested if allocation never fails.

So, I mostly did this to improve code coverage.

[catenacyber]

By the way @ktmf01 I tried nallocfuzz on Flac, and it looks like it quickly errors because the targets do not handle allocation failures