A collection of scripts to assess the accuracy and speed of different FFT implementations available in the Python packages NumPy, SciPy and pyfftw.
I only tested the code under Linux.
I tried to follow the benchmark methodology of the FFTW authors (http://www.fftw.org/benchfft/). The forward error of the FFT is calculated by comparing against a high-precision calculation of the DFT (implemented using gmpy2). Additionally, for larger array sizes the "identity error", i.e., the error between x
and ifft(fft(x))
is calculated. Note that a FFT implementation may have a large forward error while still reconstructing the original array very well.
Most configuration options can be set in config.py
. You can re-calculate the high-accuracy DFTs by by running
python prepare_accuracy_benchmark.py
The accuracy benchmark can be performed by running
python benchmark_accuracy.py numpy
where numpy
can be a string containing numpy
, scipy
or pyfftw
in which case the corresponding packages will be benchmarked and a file with the full label (e.g. numpy-1.13.3
) will be created in accuracy_results
. As a treat if you specify pocketfft
in a label (e.g. numpy-pocketfft
) it will load a numpy version from a path specified in the script (nice for testing against development versions).
The speed benchmark can be performed by running
python benchmark_speed.py numpy
where numpy
can be replaced as described above.
The evaluation plots can be created by running
python plot[_real]_accuracy_benchmark.py [save]
python plot[_real]_speed_benchmark.py [save]
I looked at the results of NumPy 1.15.0, SciPy and pyfftw. Additionally I created a version of NumPy with the proposed pocketfft library (see PR at numpy/numpy#11888) and included it under the label 'numpy-pocketfft'.
Complex FFT Accuracy Real FFT Accuracy Complex FFT Speed Real FFT Speed