|
4 | 4 |
|
5 | 5 |
|
6 | 6 |
|
7 |
| -[EvoMaster](http://evomaster.org) Benchmark (EMB): |
| 7 | + |
| 8 | +Web Fuzzing Dataset (WFD): |
8 | 9 | a set of web/enterprise applications for scientific research in Software Engineering.
|
9 | 10 |
|
10 | 11 | We collected several different systems running on the JVM, in different programming languages such as Java and Kotlin.
|
11 | 12 | In this documentation, we will refer to these projects as System Under Test (SUT).
|
12 | 13 | Currently, the SUTs are either _REST_, _GraphQL_ or _RPC_ APIs.
|
13 | 14 |
|
14 |
| -For each SUT, we implemented _driver_ classes, which can programmatically _start_, _stop_ and _reset_ the state of SUT (e.g., data in SQL databases). |
15 |
| -As well as enable setting up different properties in a _uniform_ way, like choosing TCP port numbers for the HTTP servers. |
16 |
| -If a SUT uses any external services (e.g., a SQL database), these will be automatically started via Docker in these driver classes. |
17 |
| - |
| 15 | +This dataset was previously known as EMB. It was rebranded into WFD since version 4.0.0. |
18 | 16 |
|
19 | 17 | This collection of SUTs was originally assembled for easing experimentation with the fuzzer called [EvoMaster](http://evomaster.org).
|
20 | 18 | However, finding this type of application is not trivial among open-source projects.
|
21 | 19 | Furthermore, it is not simple to sort out all the technical details on how to set these applications up and start them in a simple, uniform approach.
|
22 | 20 | Therefore, this repository provides the important contribution of providing all these necessary scripts for researchers that need this kind of case study.
|
23 | 21 |
|
24 |
| -**NOTE**: version 1.6.1 was last one in which we still updated drivers for JavaScript and C\#. Those SUTs are not built anymore by default, and latest versions of *EvoMaster* might not work on those old drivers. Updating drivers for different programming languages (and re-implement white-box heuristics) is a massive amount of work, which unfortunately has little to no value for the scientific community (based on our experience). Those SUTs are still here in EMB to enable *black-box* experiments (and to be able to replicate old experiments), but unfortunately not for *white-box* testing with latest versions of *EvoMaster*. |
| 22 | +__Black-box Testing__. For each SUT, we provide Docker Compose scripts to start the APIs with all their needed dependencies (e.g., databases). APIs are configured with mitmproxy and JaCoCo to collect information on the fuzzing results. |
| 23 | + |
| 24 | +__White-box Testing__. For each SUT, we implemented _driver_ classes, which can programmatically _start_, _stop_ and _reset_ the state of SUT (e.g., data in SQL databases). |
| 25 | +As well as enable setting up different properties in a _uniform_ way, like choosing TCP port numbers for the HTTP servers. |
| 26 | +If a SUT uses any external services (e.g., a SQL database), these will be automatically started via Docker in these driver classes. |
| 27 | + |
| 28 | +**NOTE**: version 1.6.1 was last one in which we still updated drivers for JavaScript and C\#. Those SUTs are not built anymore by default, and latest versions of *EvoMaster* might not work on those old drivers. Updating drivers for different programming languages (and re-implement white-box heuristics) is a massive amount of work, which unfortunately has little to no value for the scientific community (based on our experience). Those SUTs are still here in WFD to enable *black-box* experiments (and to be able to replicate old experiments), but unfortunately not for *white-box* testing with latest versions of *EvoMaster*. |
25 | 29 |
|
26 | 30 |
|
27 | 31 |
|
@@ -57,7 +61,7 @@ Where possible, we tried to prioritize/sort based on number of _stars_ on GitHub
|
57 | 61 |
|
58 | 62 |
|
59 | 63 | Note that some of these open-source projects might be no longer supported, whereas others are still developed and updated.
|
60 |
| -Once a system is added to EMB, we do not modify nor keep it updated with its current version under development. |
| 64 | +Once a system is added to WFD, we do not modify nor keep it updated with its current version under development. |
61 | 65 | The reason is that we want to keep an easy to use, constant set of case studies for experimentation that can be reliably used throughout the years.
|
62 | 66 |
|
63 | 67 | The SUTs called _NCS_ (Numerical Case Study) and _SCS_ (String Case study) are artificial, developed by us.
|
@@ -228,13 +232,13 @@ For simplicity, all schemas are also available as JSON/YML files under the folde
|
228 | 232 | ## Using This Repository
|
229 | 233 |
|
230 | 234 | Due to several reasons, the software in this repository is not published as a library (e.g., on Maven and NPM).
|
231 |
| -To use EMB, you need to clone this repository: |
| 235 | +To use WFD, you need to clone this repository: |
232 | 236 |
|
233 | 237 | ```
|
234 |
| -git clone https://github.com/EMResearch/EMB.git |
| 238 | +git clone https://github.com/WebFuzzing/Dataset.git |
235 | 239 | ```
|
236 | 240 |
|
237 |
| -There are 2 main use cases for EMB: |
| 241 | +There are 2 main use cases for WFD: |
238 | 242 |
|
239 | 243 | * Run experiments with _EvoMaster_
|
240 | 244 |
|
@@ -329,7 +333,7 @@ The difference is that in External the SUT is started on a separated process, an
|
329 | 333 |
|
330 | 334 | ## Old Versions
|
331 | 335 |
|
332 |
| -The release of EMB are linked in version number with the release of EvoMaster, as EvoMaster's libraries are used in the drivers (e.g., to clean databases and configure auth info). |
| 336 | +The old releases of EMB are linked in version number with the release of EvoMaster, as EvoMaster's libraries are used in the drivers (e.g., to clean databases and configure auth info). |
333 | 337 | In the Git repository of EMB, we did tag the versions of EMB.
|
334 | 338 | See the [releases](https://github.com/EMResearch/EMB/releases) page.
|
335 | 339 | For example, to use version `X`, you can check out the Git commit
|
|
0 commit comments