Skip to content

Latest commit

 

History

History
123 lines (101 loc) · 29.2 KB

README.md

File metadata and controls

123 lines (101 loc) · 29.2 KB

Overview of Property-Based Testing Functionality

Property-based testing (PBT) frameworks come with a number of different features, but which library supports which features? For a PBT newcomer, it can be hard to tell. Strictly speaking you don't even need a PBT framework. Property-based tests can be written from scratch on a case-by-case basis using a random number generator. One can even test stateful code without a state-machine framework, e.g., as outlined here. However a framework provides reusable parts and infrastructure, thus paving the way for bigger developments, such as property-based testing automotive software against the AUTOSAR specification.

  • To test an imperative API a framework with state machine support would be nice.
  • Integrated shrinking can be a nice feature for bigger developments where writing custom shrinkers may be out of the question.
  • ...

This overview is to help myself keep track. It has been compiled over a number of years teaching PBT. As features are gradually added to a framework the table's entries may unfortunately become outdated. YMMV.

I'll be happy to accept PRs for updating entries and adding new frameworks.

Framework functionality

Framework Language Gen. EDSL Shrinking Int. shr. State machine Par. st. mach. Cov. guidance
QuickCheck Erlang ✔️ ✔️ ✔️ ✔️ ✔️
PropEr Erlang ✔️ ✔️ ✔️ ✔️ ✔️
QuickCheck Haskell ✔️ ✔️ ( ✔️ ) ( ✔️ )
SmallCheck Haskell ✔️ 1 1
Hedgehog Haskell ✔️ ✔️ ✔️ ✔️ ✔️
Scala-Hedgehog Scala / JVM ✔️ ✔️ ✔️ ✔️ ✔️
R-Hedgehog R ✔️ ✔️ ✔️ ✔️
FSharp-Hedgehog F# / .Net ✔️ ✔️ ✔️
Hypothesis Python ✔️ ✔️ ✔️ ✔️ ( ✔️ )2 ( ✔️ ), ( ✔️ )
TSTL Python 3 ✔️ ✔️ ✔️ ✔️
ScalaCheck Scala / JVM ✔️ ✔️ ✔️ ✔️
gopter Go ✔️ ✔️ ✔️
Rapid Go ✔️ ✔️ ✔️ ✔️
propCheck Kotlin ✔️ ✔️ ✔️ ✔️ ✔️
FsCheck F# / .Net ✔️ ✔️ ✔️
CsCheck C# / .Net ✔️ ✔️ ✔️ ✔️ ✔️
fast-check JS / TS ✔️ ✔️ ✔️ ✔️ ( ✔️ )2
QCheck OCaml ✔️ ✔️ ✔️4 ✔️ ✔️
Crowbar OCaml ✔️ ( ✔️ )5 ✔️
Monolith OCaml ✔️ ( ✔️ )5 ✔️
Base_quickcheck OCaml ✔️ ✔️
Popper OCaml ✔️ ✔️ ✔️
junit-quickcheck Java ✔️ ✔️ ( ✔️ )6
QuickTheories Java ✔️ ✔️ ✔️ ( ✔️ ) ( ✔️ ) ✔️
jqwik Java ✔️ ✔️ ✔️ ✔️
quickcheck Rust ✔️ ✔️
propcheck Rust ✔️ ✔️ ✔️
Fox Obj.C / Swift ✔️ ✔️ ✔️ ✔️ ( ✔️ )
SwiftCheck Swift ✔️ ✔️
RapidCheck C++ ✔️ ✔️ ✔️ ( ✔️ )
test.check Clojure ✔️ ✔️ ✔️ ? ?
Lua-QuickCheck Lua ✔️ ✔️ ✔️ ✔️
theft C ( ✔️ ) ✔️ ( ✔️ ) ( ✔️ )
DeepState C / C++ ( ✔️ ) ✔️ ✔️ ✔️
Echidna Solidity / EVM ✔️ ✔️ ✔️ ✔️
...

Legend:

  • Language denotes the frameworks's language or platform
  • Generator EDSL denotes whether the framework has an expressive, embedded domain-specific language for programming custom generators (int, list, map, ...)
  • Shrinking denotes whether the framework has built-in support for reducing counterexamples.
  • Integrated shrinking denotes whether a shrinker automatically preserves any invariants of an EDSL-based custom generator (sorted lists, non-empty array, valid JSON, ...)
  • State machine denotes whether the framework has a state-machine library for model-based testing.
  • Parallel state machine denotes whether the framework supports parallel state-machine testing for race conditons, etc.
  • Coverage guidance denotes whether the framework's generators (and shrinkers) can be guided by code coverage information obtained via instrumentation.

Footnotes

Background:

The term property-based testing seems to originate from 'Property-Based Testing; A New Approach to Testing for Assurance' by Fink and Bishop (SE Notes 1997). The approach was popularized as an embedded domain-specific language in 'QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs' by Claessen and Hughes (ICFP 2000) which inspired ports to many other languages.

Integrated shrinking is explained in more detail in

State machines to test protocols and systems with state are described in

Parallel state-machine tests for race conditions were later introduced in

Additional resources: