EvoMaster: A Tool For Automatically Generating System-Level Test Cases

Summary

EvoMaster (www.evomaster.org) is the first (2016) open-source AI-driven tool that automatically generates system-level test cases for web/enterprise applications. This is related to Fuzzing. In particular, EvoMaster can fuzz APIs such as REST, GraphQL and RPC. Not only EvoMaster can generate inputs that find program crashes, but also it generates small effective test suites (e.g., in Python, JavaScript and Java/Kotlin JUnit format) that can be used for regression testing.

EvoMaster is an AI driven tool. In particular, internally it uses an Evolutionary Algorithm and Dynamic Program Analysis to be able to generate effective test cases. The approach is to evolve test cases from an initial population of random ones, trying to maximize measures like code coverage and fault detection. EvoMaster uses several kinds of AI heuristics to improve performance even further, building on decades of research in the field of Search-Based Software Testing.

1-Minute Example

On a console, copy&paste the following (requires Docker installed). It will fuzz the PetClinic example API from Swagger, for 30 seconds, as shown in the following video.

docker run -v "$(pwd)/generated_tests":/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

Using Docker on Different Shells

Note that, depending on which shell and operating system you are using, you might need slightly different commands when mounting folders with the -v option.

For example, if run in a MSYS shell on Windows like Git Bash, there is the need of an extra / before the $.

docker run -v "/$(pwd)/generated_tests":/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

If you are rather using a Command Prompt (Cmd.exe) terminal, you need to use %CD% instead of $(pwd) to refer to the current folder:

docker run -v %CD%/generated_tests:/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

On the other hand, on a PowerShell you need ${PWD}:

docker run -v ${PWD}/generated_tests:/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

Troubleshooting

If you encounter issues running the command:

Ensure Docker is installed and running.
Check that you have the correct rights/permissions to mount the specified volume.
Consult the Docker documentation for your shell environment for specific syntax requirements.

Generated Output

Once the command is executed, you can inspect the generated files under generated_tests folder.

Note, since version 4.0.0, now EvoMaster by default also creates an interactive web report.

Key features

Web APIs: At the moment, EvoMaster can generate test cases for REST, GraphQL and RPC (e.g., gRPC and Thrift) APIs.
Free: this is an open-source project, with funding from public research (e.g., professors, postdocs and PhD students). There is no monetary cost involved in using this tool, or need to use any paid external services (e.g., external LLM APIs). If you are a resident in EU, Norway, Argentina, China, or any associate country involved with ERC, thanks for your tax money supporting this project.
In-house, no telemetry: currently there is no telemetry in use in EvoMaster, and it does not require to connect to any external service on internet, besides the tested application. AFAIK, it can be run in-house without any worry of leaking any IP to external parties (if not, please open a new issue to tell us how that could happen, and we will try to fix it).
Black-Box testing mode: can run on any API (regardless of its programming language, e.g., Python and Go). However, results for black-box testing will be worse than white-box testing (e.g., due to lack of code analysis). Default test case output is in Python, but other formats are available as well.
White-Box testing mode: can be used for APIs compiled to JVM (e.g., Java and Kotlin). EvoMaster analyses the bytecode of the tested applications, and uses several heuristics such as testability transformations and taint analysis to be able to generate more effective test cases. We support JDK 8 and the major LTS versions after that (currently JDK 21). Might work on other JVM versions, but we provide NO support for it. Note: there was initial support for other languages as well, like for example JavaScript/TypeScript and C#, but they were not in a stable, feature-complete state. The support for those languages for white-box testing has been dropped, at least for the time being.
Installation: we provide installers for the main operating systems: Windows (.msi), OSX (.dmg) and Linux (.deb). We also provide an uber-fat JAR file. To download them, see the Release page. Release notes are present in the file release_notes.md. If you are using the uber-fat JAR, it should work with any major LTS version (from JDK 8 on). Whereas for the client library, needed for white-box testing, we will support JDK 8 likely for a long, long while, be warned that future versions of the executable JAR might start to require higher versions of the JDK in a non-so-distant future. If that is going to be higher than your current version of the JVM, if you cannot upgrade or have 2 different JDKs on your machine, then you should not use the uber-jar but rather one of the installers. When you use one of the installers, keep in mind that currently they do not update the PATH variable. This needs to be done manually, see documentation. Also keep in mind we have not paid the Microsoft/Apple Tax. This means that your operating system by default will block the installation, stating it cannot verify it is not a malware. But the block can be bypassed.
Docker: EvoMaster is now released via Docker as well, under webfuzzing/evomaster on Docker Hub. For more information on how to use EvoMaster via Docker, see documentation.
GitHub Action: it is possible to run EvoMaster in GitHub Actions, as part of Continuous Integration, by using the following custom action (which is in a different GitHub repository).
Hardware: although state-of-the-art AI techniques are used, there is no major hardware requirement to be able to run EvoMaster. It will work even on old laptops. The main computational bottleneck is running the tested applications, and making network calls to them.
State-of-the-art: an independent study (2022), comparing 10 fuzzers on 20 RESTful APIs, shows that EvoMaster gives the best results. Another independent study (2024) done by a different research group confirms these results.
Schema: REST APIs must provide a schema in OpenAPI format. We support versions 2.0, 3.0 and 3.1. Unfortunately, support for version 3.2 is currently on hold due to swagger-parser.
Output: the tool generates JUnit (version 4 or 5) tests, written in either Java or Kotlin, as well as test suites in Python and JavaScript. For a complete list, see the documentation for the CLI parameter --outputFormat. Some examples are: PYTHON_UNITTEST, KOTLIN_JUNIT_5, JAVA_JUNIT_4 and JS_JEST. Note that the generated tests rely on third-party libraries (e.g., to make HTTP calls). These will need to be setup in your projects, see documentation.
Web Report: besides generating executable tests in different programming language, an interactive _i

EvoMaster

Install / Use

README