benford-criterion-artifact

WISA 2026 artifact for the Benford's-law stopping criterion for fuzzing. It contains the fuzzing setups, the raw muta-byte data from the paper's campaigns, and the scripts that turn that data into Figures 1-2 and Tables 1-2 of the paper.

Two benchmarks are covered: UniFuzz (11 programs) and FuzzBench (19 programs). Every campaign is AFL++ with muta-byte logging, 14 days, 10 trials.

Layout

benchmarks/       how each fuzzing campaign was set up (Docker)
output/artifact/  the paper's per-trial muta-byte/MAD data - never written to
output/new/       everything a fresh run produces
figs/             Figure 1, Figure 2, Table 1, Table 2
scripts/          everything in between

A fresh run only ever writes under output/new/; the analysis reads output/artifact/ by default, or output/new/ with DATA=new.

Getting the data

output/artifact/ is ~11 GB, too large for git, so it is empty here. The complete artifact - these scripts plus that data - is archived on Zenodo: 10.5281/zenodo.21869385.

$ tar xJf benford-criterion-artifact.tar.xz
$ mv benford-criterion-artifact/output/artifact/* <this-clone>/output/artifact/

The unpacked archive is self-contained, so working inside it directly also works.

Regenerating the figures and tables

output/artifact/ already holds the paper's data, so no fuzzing is needed:

$ ./scripts/make_figs.sh          # Fig 1, Fig 2, Table 1, Table 2 -> figs/
$ ./scripts/make_figs.sh tables   # tables only (no Docker/R needed)

Figures 1 and 2 are drawn by the paper's own R scripts inside a container built from scripts/Dockerfile.analysis, so R is not needed on the host.

Running the fuzzing experiments

Both scripts build their Docker images and take an optional list of programs; with no arguments they run the full set used in the paper (14 days x 10 trials each).

$ ./run_unifuzz.sh                  # 11 UniFuzz programs
$ ./run_unifuzz.sh objdump jq       # only these

$ ./run_fuzzbench.sh                # 19 FuzzBench benchmarks
$ ./run_fuzzbench.sh re2_fuzzer     # only this one

No third-party fork is required. Each experiment pins the upstream commit it was built from and carries its delta as a patch:

	upstream	version	patch
UniFuzz	AFLplusplus	++4.10a	benchmarks/unifuzz/AFLplusplus/muta-byte.patch
FuzzBench	google/fuzzbench	v4.32c	benchmarks/fuzzbench/benford-overlay.patch

Both patches add the same thing to AFL++: a per-test-case count of mutated bytes, appended to introspection.txt, plus total_gen in plot_data. The FuzzBench overlay additionally carries the runner change that persists introspection.txt and type: bug on the benchmark definitions.

Extracting muta-byte data

This step is shared by both benchmarks - it runs the paper's parse.py over each trial's plot_data + introspection.txt:

$ ./scripts/extract_mad.sh unifuzz
$ PARSER=parse_fast.py ./scripts/extract_mad.sh fuzzbench

parse_fast.py is a vectorized rewrite of parse.py for the ~800 GB of FuzzBench logs; scripts/verify_parse.py <trial_dir> proves the two agree byte for byte.

Each trial becomes output/new/<bench>/<prog>-<trial>.csv:

Time,corpus_count,total_gen,edges_found,1,2,3,4,5,6,7,8,9,MAD
61, 424, 30339, 1084,5247,9342,5474,3852,2962,1411,747,783,521,0.05240499579243556

i.e. the cumulative first-digit histogram of muta-bytes and its MAD at every plot_data checkpoint. The step also keeps each trial's plot_data under output/new/plot_data/, which Table 2 needs for the crash signal. It never touches output/artifact/, so the paper's data stays intact; rebuild the figures from your own run with DATA=new ./scripts/make_figs.sh.

Analysis

Output	Scripts
Figure 1 (first-digit distributions)	fd_distribution.py -> distribution.R
Figure 2 (coverage and MAD over time)	fig2_input.py -> correlation.R
Table 1 (test cases, final MAD)	make_table1.py (aggregates the MAD CSVs)
Table 2 (stopping criteria)	parse_crash.py -> sat-{edges,crashes}_{cov,bug}.py -> make_table2.py

parse.py, parse_crash.py, fd_distribution.py, the four sat-*.py, distribution.R and correlation.R are the analysis code the paper's numbers were computed with. Three small drivers fill the steps that code leaves to the user: make_table1.py and make_table2.py average the ten trials and emit LaTeX, and fig2_input.py projects a trial's MAD CSV onto the three columns correlation.R reads. parse_fast.py is the scale helper above.

python3 scripts/make_table2.py --verify prints a cell-by-cell diff against the published table. Table 1 reproduces all 30 rows and Table 2 all 66 cells exactly.

Stopping criteria (Table 2)

All three criteria skip a 24 h warm-up, then, per trial:

• Benford's law-based: stop at the first sample with MAD <= 0.012; if that never happens, the campaign runs to the end.
• Coverage-based / crash-based: stop at the first sample whose edge coverage / saved_crashes has been unchanged for at least t, averaged over the 12 thresholds t = 2, 4, ..., 24 h.

A bug counts once its CVE-matched time-to-exposure (output/artifact/bugs/) has elapsed.

Notes

• File stems follow the fuzzer's own names (nm-new, libxml); the table generators map them to the paper's labels (nm, libxml2).
• SLoC in Table 1 is quoted from the paper; no claim depends on that column.

Citation

You can cite our paper with the following bibtex entry. The page field will be updated later.

@INPROCEEDINGS{lee:wisa:2026,
  author = {Jungwoo Lee and Haeun Lee and Sangjun Park and Sang Kil Cha},
  title = {Practical Fuzzing Saturation Detection with Benford's Law},
  booktitle = {Proceedings of the World Conference on Information Security Applications},
  year = 2026
}