TestSpark

Table of contents

• Description
• Installation
• Usage
• TestSpark for JetBrains employees
• Contribution
• Licence

Description

<!-- Plugin description -->

TestSpark is a plugin for generating unit tests. TestSpark natively integrates different AI-based test generation tools and techniques in the IDE.

TestSpark currently supports three test generation strategies:

<ul> <li>LLM-based test generation (using <a href="https://openai.com">OpenAI</a>, HuggingFace, <a href="https://ai.google/">Google AI</a>, and JetBrains internal AI Assistant platform)</li> <li>Local search-based test generation (using <a href="https://www.evosuite.org">EvoSuite</a>)</li> <li>Symbolic execution-based test generation (using <a href="https://github.com/vorpal-research/kex">Kex</a>)</li> </ul> <h4>LLM-based test generation</h4> <p>For this type of test generation, TestSpark sends request to different Large Language Models. Also, it automatically checks if tests are valid before presenting it to users.</p> <p>This feature needs a token from OpenAI, HuggingFace, Google AI, or the AI Assistant platform.</p> <ul> <li>Supports Java and Kotlin.</li> <li>Generates unit tests for capturing failures.</li> <li>Generate tests for Java classes, methods, and single lines.</li> </ul> <h4>Local search-based test generation</h4> <p>For this type of test generation, TestSpark uses <a href="https://www.evosuite.org">EvoSuite</a>, which is the most powerful search-based local test generator available for Java. </p> <ul> <li>Supports up to Java 11.</li> <li>Generates tests for different test criteria: line coverage, branch coverage, I/O diversity, exception coverage, mutation score.</li> <li>Generates unit tests for capturing failures.</li> <li>Generate tests for Java classes, methods, and single lines.</li> </ul> <h4>Symbolic execution-based test generation</h4> <p>For this type of test generation, TestSpark uses <a href="https://github.com/vorpal-research/kex">Kex</a>, supporting symbolic execution for Java Byte Code. </p> <ul> <li>Supports up to Java 8 and upwards.</li> <li>Powered by SMT solvers, it supports really high coverages given larger time frames.</li> <li>Generated test cases are however not very readable (there are plans to automatically refactor with the help of LLMs).</li> <li>Generates tests for Java classes and methods.</li> </ul> <p>Initially implemented by <a href="https://www.ciselab.nl">CISELab</a> at <a href="https://se.ewi.tudelft.nl">SERG @ TU Delft</a>, TestSpark is currently developed and maintained by the <a href="https://lp.jetbrains.com/research/software-testing/">Software Testing Research team at JetBrains Research</a>.</p> <p>If you are using TestSpark as part of your research, please cite it as follows:</p>

@inproceedings{DBLP:conf/icse/SapozhnikovOPKD24,
  author       = {Arkadii Sapozhnikov and
                  Mitchell Olsthoorn and
                  Annibale Panichella and
                  Vladimir Kovalenko and
                  Pouria Derakhshanfar},
  title        = {TestSpark: IntelliJ IDEA's Ultimate Test Generation Companion},
  booktitle    = {Proceedings of the 2024 {IEEE/ACM} 46th International Conference on
                  Software Engineering: Companion Proceedings, {ICSE} Companion 2024,
                  Lisbon, Portugal, April 14-20, 2024},
  pages        = {30--34},
  publisher    = {{ACM}},
  year         = {2024},
  url          = {https://doi.org/10.1145/3639478.3640024},
  doi          = {10.1145/3639478.3640024},
  timestamp    = {Sun, 19 Jan 2025 13:14:42 +0100},
  biburl       = {https://dblp.org/rec/conf/icse/SapozhnikovOPKD24.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

<span style="color:crimson; font-size:150%; font-weight:bold"> DISCLAIMER </span>

<span style="color:crimson; font-size:150%; font-weight:bold">TestSpark is currently designed to serve as an experimental tool.</span> <span style="color:crimson; font-size:150%; font-weight:bold">Please keep in mind that tests generated by TestSpark are meant to augment your existing test suites. They are not meant to replace writing tests manually.</span>

If you are running the plugin for the first time, checkout the Settings section.

<!-- Plugin description end -->

Installation

• Using IDE built-in plugin system:

  <kbd>Settings/Preferences</kbd> > <kbd>Plugins</kbd> > <kbd>Marketplace</kbd> > <kbd>Search for "TestSpark"</kbd> > <kbd>Install Plugin</kbd>

• Manually:

  Download the latest release and install it manually using <kbd>Settings/Preferences</kbd> > <kbd>Plugins</kbd> > <kbd>⚙️</kbd> > <kbd>Install plugin from disk...</kbd>

Usage

<!-- How to use the plugin? What are the limitations? Are there any shortcuts? -->

• Generating Tests
• Working with Test Cases
• Coverage
• Integrating tests into the project
• Settings
• Disable K2 for Kotlin Test Generation
• Telemetry

Generating Tests

To initiate the generation process, right-click on the part of the code for which tests need to be generated and select TestSpark.

Main Page

After that, a window will open where users need to configure generation settings.

Firstly users need to select the test generator (LLM-based test generator or EvoSuite, which is a Local search-based test generator).

Also, in this window, it is necessary to select the part of the code for which tests need to be generated. The selection consists of no more than three items -- class/interface, method/constructor (if applicable), line (if applicable).

After clicking the Next button, the plugin provides the opportunity to configure the basic parameters of the selected generator. Advanced parameter settings can be done in Settings. All settings, both in Settings and in this window, are saved, so you can disable the ability to configure generators before each generation process to perform this process more quickly.

LLM Setup Page

In the case of LLM, two additional pages are provided for basic settings.
In the first page, users configure LLM Platform, LLM Token, LLM Model, LLM JUnit version, and Prompt Selection. More detailed descriptions of each item can be found in Settings.

LLM Samples Page

After that, in the next page, you can provide some test samples for LLM.
Tests can be entered manually.

Also, tests can be chosen tests from the current project.

Test Cases can be modified, reset to their initial state, and deleted.

EvoSuite Setup Page

For EvoSuite, you need to enter the local path to Java 11 and select the generation algorithm, after which the generation process will start.

Generation Process

After configuring the test generators, click the OK button, after which the generation process will start, and a list of generated test cases will appear on the right side of the IDE.

![Generated tests dark](readme-images/pngs/generating-tests/Gene