Defects4J -- version 3.0.1

Defects4J is a collection of reproducible bugs and a supporting infrastructure with the goal of advancing software engineering research.

Contents of Defects4J

The projects

Defects4J contains 854 bugs (plus 10 deprecated bugs) from the following open-source projects:

| Identifier | Project name | Number of active bugs | Active bug ids | Deprecated bug ids (*) | |-----------------|----------------------------|----------------------:|---------------------|-------------------------| | Chart | jfreechart | 26 | 1-26 | None | | Cli | commons-cli | 39 | 1-5,7-40 | 6 | | Closure | closure-compiler | 174 | 1-62,64-92,94-176 | 63,93 | | Codec | commons-codec | 18 | 1-18 | None | | Collections | commons-collections | 28 | 1-28 | None | | Compress | commons-compress | 47 | 1-47 | None | | Csv | commons-csv | 16 | 1-16 | None | | Gson | gson | 18 | 1-18 | None | | JacksonCore | jackson-core | 26 | 1-26 | None | | JacksonDatabind | jackson-databind | 110 | 1-64,66-88,90-112 | 65,89 | | JacksonXml | jackson-dataformat-xml | 6 | 1-6 | None | | Jsoup | jsoup | 93 | 1-93 | None | | JxPath | commons-jxpath | 22 | 1-22 | None | | Lang | commons-lang | 61 | 1,3-17,19-24,26-47,49-65 | 2,18,25,48 | | Math | commons-math | 106 | 1-106 | None | | Mockito | mockito | 38 | 1-38 | None | | Time | joda-time | 26 | 1-20,22-27 | 21 |

* Due to behavioral changes introduced in newer Java versions, some bugs are no longer reproducible. Hence, Defects4J distinguishes between active and deprecated bugs:

• Active bugs can be accessed through active-bugs.csv.

• Deprecated bugs are removed from active-bugs.csv, but their metadata is retained in the project directory.

• Deprecated bugs can be accessed through deprecated-bugs.csv, which also details when and why a bug was deprecated.

We do not re-enumerate active bugs because publications using Defects4J artifacts usually refer to bugs by their specific bug id.

The bugs

Each bug has the following properties:

• Issue filed in the corresponding issue tracker, and issue tracker identifier mentioned in the fixing commit message.
• Fixed in a single commit.
• Minimized: the Defects4J maintainers manually pruned out irrelevant changes in the commit (e.g., refactorings or feature additions).
• Fixed by modifying the source code (as opposed to configuration files, documentation, or test files).
• A triggering test exists that failed before the fix and passes after the fix -- the test failure is not random or dependent on test execution order.

The (b)uggy and (f)ixed program revisions are labelled with <id>b and <id>f, respectively (<id> is an integer).

Reproducibility

Java version

All bugs have been reproduced and triggering tests verified, using Java 11 (see the CI configuration for specifics). Using a different version of Java might result in unexpected failing tests and/or non-reproducible bugs.

Timezone

Defects4J generates and executes tests in the timezone America/Los_Angeles.

If you are using the bugs outside of the Defects4J framework, set the TZ environment variable to America/Los_Angeles and export it. Not setting this option results in unexpected failing tests!

Broken and Flaky Tests

Defects4J excludes broken tests (tests that reliably fail on the fixed and the buggy version) and flaky tests (tests that intermittently fail on the fixed or buggy version).

If you are using the bugs outside of the Defects4J framework, make sure to verify expected test behavior for your environment.

Metadata

We recommend using Defects4J through the provided command-line interface. All reproducibility tests within Defects4J rely on this interface. If you are using the bugs outside of the Defects4J framework, make sure to use the Defects4J export command to obtain relevant metadata such as source/test directories, classpath entries, and sets of tests.

Setting up Defects4J

Requirements

• Java 11
• Git >= 1.9
• Subversion (svn) >= 1.8
• Perl >= 5.0.12
• cpanm

Defects4J version 2.x required Java 1.8.

Defects4J version 1.x and 0.x required Java 1.7.

Perl dependencies

All required Perl modules are listed in cpanfile. On many Unix platforms, these required Perl modules are installed by default. The setup instructions immediately below install them if necessary. If you do not have cpanm installed, use cpan or a cpan wrapper to install the perl modules listed in cpanfile.

Steps to set up Defects4J

1. Clone Defects4J:

   • git clone https://github.com/rjust/defects4j

2. Initialize Defects4J (download the project repositories and external libraries, which are not included in the git repository for size purposes and to avoid redundancies):

   • cd defects4j
   • cpanm --installdeps .
   • ./init.sh

3. Add Defects4J's executables to your PATH:

   • export PATH=$PATH:"path2defects4j"/framework/bin ("path2defects4j" points to the directory to which you cloned Defects4J; it looks like "/user/yourComputerUserName/desktop/defects4j".)

4. Check installation:

   • defects4j info -p Lang

On some platforms such as Windows, you might need to use perl "fullpath"\defects4j where these instructions say to use defects4j.

Using Defects4J

Example commands

1. Get information for a specific project (commons lang):

   • defects4j info -p Lang

2. Get information for a specific bug (commons lang, bug 1):

   • defects4j info -p Lang -b 1

3. Checkout a buggy source code version (commons lang, bug 1, buggy version):

   • defects4j checkout -p Lang -v 1b -w /tmp/lang_1_buggy

4. Change to the working directory, compile sources and tests, and run tests:

   • cd /tmp/lang_1_buggy
   • defects4j compile
   • defects4j test

5. Some Defects4J commands take the project id as a command-line argument (possibly along with other arguments). Examples include info, checkout, and query. Note that info and query report information that is derived from the Defects4J metadata and do not require access to project files that are in the project's VCS.

   Other commands require a working directory, either set explicitly (-w command-line argument) or implicitly (executed from within a working directory). Examples include any command that requires access to files under version control, including source code and build files. This includes all commands that build or test the code (compile, test, coverage, mutation) and commands that return version-specific information (export).

6. The scripts in framework/test/ are examples of how to use Defects4J, which you might find useful as inspiration when you are writing your own scripts that use Defects4J.

Command-line interface: defects4j command

Use framework/bin/defects4j to execute any of the following commands:

| Command | Description | |----------------|---------------------------------------------------------------------------------------------------| | info | View configuration of a specific project or summary of a specific bug | | env | Print the environment of defects4j executions | | checkout | Checkout a buggy or a fixed project version | | compile | Compile sources and developer-written tests of a buggy or a fixed project version | | test | Run a single test method or a test suite on a buggy or a fixed project version | | mutation | Run mutation analysis on a buggy or a fixed project version | | [covera