Awfy (Are We Fast Yet)

A CLI tool for running and comparing Ruby benchmarks across different implementations, runtimes (MRI/YJIT), and git branches or commits.

Features

• Multiple Benchmark Types

  • IPS benchmarks (using benchmark-ips)
  • Memory profiling (using memory_profiler)
  • CPU profiling (using stackprof)
  • Flamegraph generation (using vernier)
  • YJIT statistics

• Rich Comparison Features

  • Compare multiple implementations
  • Compare across git branches or commit ranges
  • Compare with/without YJIT
  • Generate summary reports

• Persist results

  • SQLite storage (default)
  • JSON file storage

Example output:

Installation

Add to your application:

group :development, :test do
  gem "awfy", require: false
end

Or install directly:

gem install awfy

Quick Start

1. Create a benchmark suite directory:

mkdir -p benchmarks/tests

2. Create a setup file (optional):

# benchmarks/setup.rb
require "awfy"
require "json"  # Add any dependencies your benchmarks need

# Setup test data or helper methods
SAMPLE_DATA = { "name" => "test", "values" => [1, 2, 3] }.freeze

def create_test_object
  SAMPLE_DATA.dup
end

3. Write your first benchmark:

# benchmarks/tests/json_parsing.rb
Awfy.group "JSON" do
  report "#parse" do
    # Setup test data
    json_string = SAMPLE_DATA.to_json
    
    # Benchmark standard library as control
    control "JSON.parse" do
      JSON.parse(json_string)
    end
    
    # Benchmark your implementation
    test "MyJSONParser" do
      MyJSONParser.parse(json_string)
    end
  end
end

4. Run the benchmark:

For example:

# Run IPS benchmark
bundle exec awfy ips
# or more explicitly:
# bundle exec awfy ips start JSON "#parse"

# Compare with another branch
bundle exec awfy ips --compare-with=main

# Run across a commit range
bundle exec awfy ips start --commit-range="HEAD~5..HEAD" --runner=commit_range

# Run across commits with a specific baseline
bundle exec awfy ips start --commit-range="HEAD~5..HEAD" --runner=commit_range --control-commit=HEAD~5

# Run without YJIT
bundle exec awfy ips --runtime=mri

# Run benchmark in parallel
bundle exec awfy ips --runner=forked

Documentation

For detailed documentation, see:

• Benchmark Suite Guide - How to write benchmarks
• Configuration Guide - Configuration options
• Command Reference - Available commands
• Advanced Usage - Advanced features
• Best Practices - Tips and guidelines

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/stevegeek/awfy.

License

Available as open source under the terms of the MIT License.