WFD

Web Fuzzing Dataset (WFD): a set of web/enterprise applications for scientific research in Software Engineering.

We collected several different systems running on the JVM, in different programming languages such as Java and Kotlin. In this documentation, we will refer to these projects as System Under Test (SUT). Currently, the SUTs are either REST, GraphQL or RPC APIs.

This dataset was previously known as EMB. It was rebranded into WFD since version 4.0.0.

This collection of SUTs was originally assembled for easing experimentation with the fuzzer called EvoMaster. However, finding this type of application is not trivial among open-source projects. Furthermore, it is not simple to sort out all the technical details on how to set these applications up and start them in a simple, uniform approach. Therefore, this repository provides the important contribution of providing all these necessary scripts for researchers that need this kind of case study.

Black-box Testing. For each SUT, we provide Docker Compose scripts (under the dockerfiles folder) to start the APIs with all their needed dependencies (e.g., databases). APIs are configured with mitmproxy and JaCoCo to collect information on the fuzzing results.

White-box Testing. For each SUT, we implemented driver classes for EvoMaster (currently the only existing white-box fuzzer for the JVM), which can programmatically start, stop and reset the state of SUT (e.g., data in SQL databases). As well as enable setting up different properties in a uniform way, like choosing TCP port numbers for the HTTP servers. If a SUT uses any external services (e.g., a SQL database), these will be automatically started via Docker in these driver classes.

NOTE: version 1.6.1 was last one in which we still updated drivers for JavaScript and C#. Those SUTs are not built anymore by default, and latest versions of EvoMaster will not work on those old drivers. Updating drivers for different programming languages (and re-implement white-box heuristics) is a massive amount of work, which unfortunately has little to no value for the scientific community (based on our experience). Those SUTs are still here in WFD to be able to replicate old experiments, but unfortunately not for white-box testing with latest versions of EvoMaster.

An old video (2023) providing some high level overview of EMB can be found here.

License

All the code that is new for this repository (e.g., the driver classes) is released under Apache 2.0 license. However, this repository contains as well sources from different open-source projects, each one with its own license, as clarified in more details beneath.

Example

To see an example of using these drivers with EvoMaster to generate test cases, you can look at this short video (5 minutes).

Citation

If you are using WFD in an academic work, you can cite the following:

O. Sahin, M. Zhang, A. Arcuri. WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing. arxiv 2509.01612

For the old version still called EMB, you can refer to:

A. Arcuri, M. Zhang, A. Golmohammadi, A. Belhadi, J. P. Galeotti, B. Marculescu, S. Seran. EMB: A Curated Corpus of Web/Enterprise Applications And Library Support for Software Testing Research. In IEEE International Conference on Software Testing, Validation and Verification (ICST), 2023.

Current Case Studies

The projects were selected based on searches using keywords on GitHub APIs, using convenience sampling. Several SUTs were looked at, in which we discarded the ones that would not compile, would crash at startup, would use obscure/unpopular libraries with no documentation to get them started, are too trivial, student projects, etc. Where possible, we tried to prioritize/sort based on number of stars on GitHub. When authors of other fuzzers used some other open-source JVM APIs in their studies, we included them here into WFD.

Note that some of these open-source projects might be no longer supported, whereas others are still developed and updated. Once a system is added to WFD, we do not modify nor keep it updated with its current version under development. The reason is that we want to keep an easy to use, constant set of case studies for experimentation that can be reliably used throughout the years.

The SUTs called NCS (Numerical Case Study) and SCS (String Case study) are artificial, developed by us. They are based on numerical and string-based functions previously used in the literature of unit test generation. We just re-implemented in different languages, and put them behind a web service.

For the RESTful APIs, each API has an endpoint where the OpenAPI/Swagger schemas can be downloaded from. For simplicity, all schemas are also available as JSON/YML files under the folder openapi-swagger.

IMPORTANT: More details (e.g., #LOCs and used databases) on these APIs can be found in this table.

Real-world APIs require authentication. How to setup authentication information, based on the current content of the initialized databases, is expressed in Web Fuzzing Commons (WFC) format. Auth configuration files can found in the auth folder.

REST: Java/Kotlin (36)

• Bibliothek (MIT), jdk_17_gradle/cs/rest/bibliothek, from https://github.com/PaperMC/bibliothek

• Blog (AGPL), jdk_8_maven/cs/rest/original/blogapi, from https://github.com/osopromadze/Spring-Boot-Blog-REST-API

• CatWatch (Apache), jdk_8_maven/cs/rest/original/catwatch, from https://github.com/zalando-incubator/catwatch

• CWA-Verification-Server (Apache), jdk_11_maven/cs/rest/cwa-verification-server, from https://github.com/corona-warn-app/cwa-verification-server

• ERC20 Rest Service (not-known license), jdk_8_gradle/cs/rest/erc20-rest-service, from https://github.com/web3labs/erc20-rest-service

• Familie Ba Sak (MIT), jdk_17_maven/cs/rest/familie-ba-sak, from https://github.com/navikt/familie-ba-sak

• Features-Service (Apache), jdk_8_maven/cs/rest/original/features-service, from https://github.com/JavierMF/features-service

• Genome Nexus (MIT), jdk_8_maven/cs/rest-gui/genome-nexus, from https://github.com/genome-nexus/genome-nexus

• Gestao Hospital (not-known license), jdk_8_maven/cs/rest-gui/gestaohospital, from https://github.com/ValchanOficial/GestaoHospital

• HTTP Patch Spring (MIT), jdk_11_maven/cs/rest/http-patch-spring, from https://github.com/cassiomolin/http-patch-spring

• Languagetool (LGPL), jdk_8_maven/cs/rest/original/languagetool, from https://github.com/languagetool-org/languagetool

• Market (MIT), jdk_11_maven/cs/rest-gui/market, from https://github.com/aleksey-lukyanets/market

• Microcks (Apache), jdk_21_maven/cs/rest-gui/microcks, from https://github.com/microcks/microcks

• NCS, jdk_8_maven/cs/rest/artificial/ncs, (not-known license, artificial numerical examples coming from different sources)

• News (LGPL), jdk_8_maven/cs/rest/artificial/news, from https://github.com/arcuri82/testing_security_development_enterprise_systems

• OCVN (MIT), jdk_8_maven/cs/rest-gui/ocvn, from https://github.com/devgateway/ocvn

• Ohsome API (AGPL-3.0), jdk_17_maven/cs/rest/ohsome-api, from https://github.com/GIScience/ohsome-api

• Payments Public API (MIT), jdk_11_maven/cs/rest/pay-publicapi, from https://github.com/alphagov/pay-publicapi

• Person Controller (Apache), jdk_21_maven/cs/rest/person-controller, from https://github.com/mongodb-developer/java-spring-boot-mongodb-starter

• Project Tracking System (not-known license), jdk_11_maven/cs/rest/tracking-system, from [https://github.com/SelimHorri/project-tracking-system-backend-app](https://github.com/SelimHorri/project-tracking-system-backen