Renaissance

Renaissance Benchmark Suite

<p align="center"> <img height="180px" src="https://github.com/renaissance-benchmarks/renaissance/raw/master/logo.png"/> </p>

The Renaissance Benchmark Suite aggregates common modern JVM workloads, including, but not limited to, Big Data, machine-learning, and functional programming. The suite is intended to be used to optimize just-in-time compilers, interpreters, GCs, and for tools such as profilers, debuggers, or static analyzers, and even different hardware. It is intended to be an open-source, collaborative project, in which the community can propose and improve benchmark workloads.

Running the suite

To run the suite, you will need to download a Renaissance Suite JAR from https://renaissance.dev/download. If you wish to build it yourself, please, consult CONTRIBUTING.md for instructions on building.

To run a Renaissance benchmark, you need to have a JRE version 11 (or later) installed and execute the following java command:

$ java -jar 'renaissance-gpl-0.16.1.jar' <benchmarks>

In the above command, <benchmarks> is the list of benchmarks that you want to run. You can refer to individual benchmarks, e.g., scala-kmeans, or a group of benchmarks, e.g., apache-spark.

The suite generally executes the benchmark's measured operation multiple times. By default, the suite executes each benchmark operation for a specific number of times. The benchmark-specific number of repetitions is only intended for quick visual evaluation of benchmark execution time, but is not sufficient for thorough experimental evaluation, which will generally need much more repetitions.

For thorough experimental evaluation, the benchmarks should be repeated for a large number of times or executed for a long time. The number of repetitions and the execution time can be set for all benchmarks using the -r or -t options. More fine-grained control over benchmark execution can be achieved by providing the harness with a plugin implementing a custom execution policy (see below for details).

Known issues

This section lists selected issues that users may encounter when running Renaissance. Because these issues cannot be simply fixed or avoided by the Renaissance harness, we provide suggested workarounds.

Possible StackOverflowError in Spark benchmarks

• Issue: Spark benchmarks may fail with a StackOverflowError on certain JVM configurations. This is a known issue with Spark applications that allocate large amounts of memory on the stack. See #375 for details.
• Workaround: The workaround is to increase the JVM stack size via the -Xss command line option to provide more stack headroom (for example, -Xss4m or more). Add this to the JVM options when invoking the benchmark harness.

Spark benchmarks may fail on Windows due to missing libraries

• Issue: Spark uses Hadoop, which loads a native library, hadoop.dll, when running on Windows. The library packaged with the Renaissance suite requires Visual C++ 2013 runtime to be present on the system. Running Spark benchmarks on a Windows system without the required runtime will fail. See #480 for details.
• Workaround: Install the appropriate runtime from the Visual C++ Redistributable Package for VS 2013 or its updated version from the linked KB 3138367.

finagle-chirper throws ConnectionFailedException on machines with many CPUs

• Issue: On machines with many CPUs the finagle-chirper benchchmark can start throwing ConnectionFailedExceptions and becomes unstable (i.e. even after warmup, adjacent iterations can have a very high variance). The problem is caused by hundreds of thousands of connections lingering in TIME_WAIT status (use netstat -n | grep TIME_WAIT | wc to verify) before they finally get closed (see #231 for more details).
• Workaround: Currently, the only known workaround is to decrease the numbers of CPUs for the benchmark with the JVM command line option -XX:ActiveProcessorCount=. E.g. on a machine with 96 cores, the benchmark reproducibly throws ConnectionFailedExceptions while running with -XX:ActiveProcessorCount=48 works just fine.

The harness may fail to remove its temporary directories

• Issue: The harness will create launcher-* and harness-* temporary directories in the working directory. When the temporary directories are on an NFS mount or a file system that does not allow deleting open files, the harness may fail to clean up and remove the temporary directories. See #464 for details.
• Workaround: When executed manually (under user supervision), the launcher-* and harness-* directories can be deleted manually after the run. Alternatively, avoid running Renaissance from a working directory on NFS (or a known problematic file system), or explicitly set the location of the scratch space via the --scratch-base <dir> command line option (for example, --scratch-base /tmp). When executed automatically (without user supervision), it is recommended to launch Renaissance using a wrapper script responsible for creating and removing the top-level temporary directory, and for running Renaissance with the corresponding --scratch-base <dir> option.

Licensing

The Renaissance Suite comes in two distributions, and is available under both the MIT license and the GPL3 license. The GPL distribution with all the benchmarks is licensed under the GPL3 license, while the MIT distribution includes only those benchmarks that themselves have less restrictive licenses.

Depending on your needs, you can use either of the two distributions.

The list below contains the licensing information (and JVM version requirements) for each benchmark.

List of benchmarks

The following is the complete list of benchmarks, separated into groups.

apache-spark

• als - Runs the ALS algorithm from the Spark ML library.
  Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• chi-square - Runs the chi-square test from Spark MLlib.
  Default repetitions: 60; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• dec-tree - Runs the Random Forest algorithm from the Spark ML library.
  Default repetitions: 40; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• gauss-mix - Computes a Gaussian mixture model using expectation-maximization.
  Default repetitions: 40; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• log-regression - Runs the Logistic Regression algorithm from the Spark ML library.
  Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• movie-lens - Recommends movies using the ALS algorithm.
  Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• naive-bayes - Runs the multinomial Naive Bayes algorithm from the Spark ML library.
  Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• page-rank - Runs a number of PageRank iterations, using RDDs.
  Default repetitions: 20; APACHE2 license, MIT distribution; Supported JVM: 11 and later

concurrency

• akka-uct - Runs the Unbalanced Cobwebbed Tree actor workload in Akka.
  Default repetitions: 24; MIT license, MIT distribution; Supported JVM: 11 and later

• fj-kmeans - Runs the K-Means algorithm using the fork/join framework.
  Default repetitions: 30; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• reactors - Runs benchmarks inspired by the Savina microbenchmark workloads in a sequence on Reactors.IO.
  Default repetitions: 10; MIT license, MIT distribution; Supported JVM: 11 and later

database

• db-shootout - Executes a shootout test using several in-memory databases.
  Default repetitions: 16; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• neo4j-analytics - Executes Neo4j graph queries against a movie database.
  Default repetitions: 20; GPL3 license, GPL3 distribution; Supported JVM: 17 and later

functional

• future-genetic - Runs a genetic algorithm using the Jenetics library and futures.
  Default repetitions: 50; APACHE2 license, MIT distribution; Supported JVM: 11 and later

• mnemonics - Solves the phone mnemonics problem using JDK streams.
  Default repetitions: 16; MIT license, MIT distribution; Supported JVM: 11 and later

• par-mnemonics - Solves the phone mnemonics problem using parallel JDK streams.
  Default repetitions: 16; MIT license, MIT distribution; Supported JVM: 11 and later

• rx-scrabble - Solves the Scrabble puzzle using the Rx streams.
  Default repetitions: 80; GPL2 license, GPL3 distribution; Supported JVM: 11 and later

• scrabble - Solves the Scrabble puzzle using JDK Streams.
  Default repetitions: 50; GPL2 license, GPL3 distribution; Supported JVM: 11 and later

scala

• dotty - Runs the Dotty compiler on a set of source code files.
  Default repetitions: 50; BSD3 license, MIT distribution; Supported JVM: 11 and later

• philosophers - Solves a variant of the dining philosophers problem using ScalaSTM.
  Default repetitions: 30; BSD3 license, MIT distribution; Supported JVM: 11 and later

• scala-doku - Solves Sudoku Puzzles using Scala collections.
  Default repetitions: 20; MIT license, MIT distribution; Supported JVM: 11 and later

• scala-kmeans - Runs the K-Means algorithm using Scala collections.
  Default