Artemis

<p align="center"> <img width="320" alt="logo" align="center" src=".github/artemis.png"> </p> <br/> <p align="center"> <a href="https://github.com/test-jitcomp/Artemis/actions" alt="Build status"> <img src="https://img.shields.io/github/actions/workflow/status/test-jitcomp/Artemis/build.yml" /> </a> <a href="https://semver.org" alt="Version"> <img src="https://img.shields.io/github/v/release/test-jitcomp/Artemis" /> </a> <a href="https://google.github.io/styleguide/javaguide.html" alt="Code style"> <img src="https://img.shields.io/badge/style-Google-blue" /> </a> <a href="https://dl.acm.org/doi/10.1145/3600006.3613140" alt="SOSP 2023"> <img src="https://img.shields.io/badge/2023-SOSP-8A2BE2" /> </a> <a href="https://opensource.org/licenses/MIT" alt="License"> <img src="https://img.shields.io/github/license/test-jitcomp/Artemis" /> </a> </p>

Artemis is a Java program mutator specifically designed to test JVM's JIT compilers. It also provides a fuzzing framework called artemi to stress-test JVM's JIT compilers. Given a set of seed Java programs, Artemis generates a set of mutants for testing. Artemis has already found 80+ JVM's JIT compiler bugs in four widely-used production JVMs: HotSpot, OpenJ9, Android Runtime, and GraalVM.

JIT compiler bug: A JIT compiler bug is a bug that otherwise won't manifest if the JIT compiler is disabled for example by the -Xint option in prevalent JVMs.

💡 Requirements

Artemis is tested and developed on Ubuntu and macOS platforms with the following software/hardware requirements.

• Java: >= 11
• Python: >= 3.9.0
• Ruby: >= 2.7.0

📦 Installation

Install Java, Python, and Ruby. For Ubuntu:

$ sudo apt update && apt install -y openjdk-11-jdk python3 python3-pip ruby-full

Download Artemis from the Releases page and unzip to a directory say /tmp/artemis:

$ unzip artemis-<version>.zip -d /tmp/artemis

Download Code Bricks from Releases page and unzip to a directory say /tmp/artemis/cbs.

$ unzip code-bricks.zip -d /tmp/artemis/cbs

Install required dependencies:

$ cd /tmp/artemis
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

🚀 Fuzzing JVMs

The artemi framework is designed to have no command line options except for a fuzzing configuration file called artemi.yaml:

1. Edit artemi.yaml. See artemi.ex.yaml for an example.

   • Do fill every option marked as <required-to-change> with correct value. But for the jvm and generator option, only fill those related despite marked as <required-to-change>.
   • Do fill with absolute paths for each path option, instead of relative paths.
   • The framework supports to test HotSpot, OpenJ9, Graal, and ART at present. Supporting other JVMs is on the way. For this option, do download (or build by yourself) the specific-version JVM you'd like to test and points *_home sub-option to the home directory of it. To build by yourself, follow these threads:
     • HotSpot: Building the JDK (17)
     • OpenJ9: Building OpenJDK Version 17 with OpenJ9
     • ART: Building Android; for this, use m build-art-host or m build-art-target in the last step to build quickly.
   • The framework supports to use Java*Fuzzer and JFuzz as the program generator at present. Supporting other program generators is on the way.
   • The framework has no other command line arguments and options except those listed in artemi.yaml.

2. Create required directories. Create out_dir and generator.out_dir as you've specified in artemi.yaml. These are output directories of artemi and the Java generator you've used, respectively.

   $ mkdir -p <out_dir> <generator.out_dir>
   

3. Run the artemi framework. The artemi framework will run in an infinite loop. You can use the shortcut CTRL+C to terminate artemi manually.

   $ python artemi.py artemi.yaml
   

4. Check detected bugs. All bugs that are detected are listed in <out_dir>/differences/diffs.csv where <out_dir>/differences/<diff_id> saves the seed, the mutant, the output of the seed, the output of the mutant, and Artemis' mutation log.

Note. Although Artemis is designed to generate syntax- and semantic-valid mutants, bugs of Artemis itself may break this. Directory <out_dir>/mutation-failures saves cases which causes Artemis to fail in mutating, and <out_dir>/compilation-failures saves cases when the mutant fails to compile.

☄️ Mutating Programs

Besides the artemi framework, Artemis itself can be used as a seperate program mutator. It takes as input a Java source file, and outputs a Java source file with mutations specifically designed to test JIT compilers. See required arguments and available options by -h.

$ java -jar artemis.jar -h

🐞 Bug Showcases

Artemis is fruitful in finding diverse bugs ranging from segmentation faults (SIGSEGV), fatal arithmetic error (SIGFPE), emergency abort (SIGABRT), assertion failures, mis-compilations, to performance issues. These bugs affect quite a few VM components. We list some of them here. More to come.

It should be noted that, to avoid flooding their issue trackers, we discussed with the corresponding VM developers and reported some difficult-to-reproduce, flaky tests into a single issue. This kept the number of bug reports under a small limit.

In addition, some bugs listed below were made internal by the respective JVM developers and are no longer publicly accessible for security reasons.

HotSpot

• JDK-8287223: P3, C1, Assertion Failure, Inlining
• JDK-8288198: P2, C2, Assertion Failure, Ideal Graph Building
• JDK-8288734: P4, C2, Assertion Failure, Ideal Graph Building
• JDK-8305429: P4, C2, Assertion Failure, Ideal Graph Building
• JDK-8290781: P3, C2, Segmentation Fault, Ideal Loop Optimization
• JDK-8292766: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8294217: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8294433: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8294413: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8290778: P2, C2, Segmentation Fault, Ideal Loop Optimization
• JDK-8288558: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8288198: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8288106: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8305797: P4, C2, Assertion Failure, Ideal Loop Optimization
• JDK-8288187: P4, C2, Assertion Failure, Global Constant Propagation
• JDK-8288206: P4, C2, Assertion Failure, Global Value Numbering
• JDK-8288587: P4, C2, Assertion Failure, Global Value Numbering
• JDK-8287217: P4, C2, Assertion Failure, Global Value Numbering
• JDK-8293996: P4, C2, Assertion Failure, Global Value Numbering
• JDK-8288204: P3, C2, Assertion Failure, Global Value Numbering
• JDK-8288392: P4, C2, Assertion Failure, Escape Analysis
• JDK-8288559: P4, C2, Assertion Failure, Register Allocation
• JDK-8290862: P4, C2, Segmentation Fault, Register Allocation
• JDK-8290776: P3, C2, Segmentation Fault, Code Generation
• JDK-8288560: P4, C2, Assertion Failure, Code Generation
• JDK-8289043: P3, C2, Assertion Failure, Code Generation
• JDK-8305428: P4, C2, Assertion Failure, Code Generation
• JDK-8288190: P2, C2, Segmentation Fault, Code Execution
• JDK-8290789: P3, C2, Segmentation Fault, Code Execution
• JDK-8290864: P4, C2, Segmentation Fault, Code Execution
• JDK-8288975: P3, C2, Mis-compilation, Global Code Motion
• JDK-8290360: P3, C2, Performance Issue

OpenJ9

• 15332: Assertion Failure, Local Value Propagation
• 15311: Segmentation Fault, Global Value Propagation
• 15364: Segmentation Fault, Global