`testAllMethods` option for the `fuzz` goal would be nice to have

[yegor256]

I have a class with a number of methods annotated with @Fuzz. Currently, I have to create multiple <execution> elements for the jqf-maven-plugin, one per each method. Would be more convenient to have testAllMethods property, which would simply instruct the plugin to run all methods, which are @Fuzz-annotated.

[rohanpadhye]

Interesting. This isn't something that I've seen anyone need before because fuzzing is typically run with user-defined entry points. But, I see the use case if using JQF to fuzz all methods in a test suite / test class for a limited time each. I can add this to my TODO list. It will probably be easiest to implement using the existing testMethod parameter set to a special value * or similar.

[yegor256]

@rohanpadhye maybe you can also do the same with the testClassName? The motivation is similar: I simply want your plugin to scan the entire code base and run all @Fuzz-annotated methods.

[rohanpadhye]

Doing this for all classes is trickier because it is not trivial to find all test classes unless they are explicitly loaded. JUnit has some functionality to scan the classpath for classes named *Test and then run them, but the public API that allows this will run all JUnit tests in these classes, not just those annotated with @Fuzz. This can already be done today by just running the Maven surefire plugin, which runs all JUnit tests including those annotated with @Fuzz.

Having a scanner that only searches for fuzz entry points but not other JUnit tests will require cloning a lot of low-level functionality from JUnit, which I'd like to avoid if possible.

It might be easier to achieve the goal of testing everything by just relying on the surefire plugin and using some wildcards / naming scheme to filter tests to only fuzzable ones and not basic unit tests. However, today this will run the @Fuzz tests with black-box input generation, since it is not possible to get JQF's coverage instrumentation through Surefire. Perhaps it is worth figuring out how to get coverage instrumentation through Surefire, so that the logic for running/selecting tests is not JQF's problem to solve (with duplicated logic).

[yegor256]

@rohanpadhye how about adding another argument to the plugin: testPackage. When the package name is provided, you do this, in order to find all classes that need to be dealt with (using Reflections):

new Reflections(pkg).getTypesAnnotatedWith(RunWith.class)

Then, you simply check the value of the RunWith. If it's set to edu.berkeley.cs.jqf.fuzz.JQF.class, you have a class to process.