Property-Based Testing in Ruby

A property-based testing tool for Ruby with experimental features that allow you to run test cases in parallel.

PBT stands for Property-Based Testing.

As for the results of the parallelization experiment, please refer the talk at RubyKaigi 2024: Unlocking Potential of Property Based Testing with Ractor.

What's Property-Based Testing?

Property-Based Testing is a testing methodology that focuses on the properties a system should always satisfy, rather than checking individual examples. Instead of writing tests for predefined inputs and outputs, PBT allows you to specify the general characteristics that your code should adhere to and then automatically generates a wide range of inputs to verify these properties.

The key benefits of property-based testing include the ability to cover more edge cases and the potential to discover bugs that traditional example-based tests might miss. It's particularly useful for identifying unexpected behaviors in your code by testing it against a vast set of inputs, including those you might not have considered.

For a more in-depth understanding of Property-Based Testing, please refer to external resources.

• Original ideas
  • Property-based testing of privileged programs (1994)
  • Property-based testing: a new approach to testing for assurance (1997)
  • QuickCheck: a lightweight tool for random testing of Haskell programs (2000)
• Rather new introductory resources
  • Fred Hebert's book Property-Based Testing With PropEr, Erlang and Elixir.
  • fast-check - Why Property-Based?

Installation

Add this line to your application's Gemfile and run bundle install.

gem 'pbt'

Of course you can install with gem install pbt.

Basic Usage

Simple property

# Let's say you have your own sort method.
def sort(array)
  return array if array.size <= 2 # Here's a bug! It should be 1.
  pivot, *rest = array
  left, right = rest.partition { |n| n <= pivot }
  sort(left) + [pivot] + sort(right)
end

Pbt.assert do
  # The given block is executed 100 times with different arrays with random numbers.
  # Besides, if you set `worker: :ractor` option to `assert` method, it runs in parallel using Ractor.
  Pbt.property(Pbt.array(Pbt.integer)) do |numbers|
    result = sort(numbers)
    result.each_cons(2) do |x, y|
      raise "Sort algorithm is wrong." unless x <= y
    end
  end
end

# If the method has a bug, the test fails and it reports a minimum counterexample.
# For example, the sort method doesn't work for [0, -1].
#
# Pbt::PropertyFailure:
#   Property failed after 23 test(s)
#   seed: 43738985293126714007411539287084402325
#   counterexample: [0, -1]
#   Shrunk 40 time(s)
#   Got RuntimeError: Sort algorithm is wrong.

Explain The Snippet

The above snippet is very simple but contains the basic components.

Runner

Pbt.assert is the runner. The runner interprets and executes the given property. Pbt.assert takes a property and runs it multiple times. If the property fails, it tries to shrink the input that caused the failure.

Property

The snippet above declared a property by calling Pbt.property. The property describes the following:

1. What the user wants to evaluate. This corresponds to the block (let's call this predicate) enclosed by do end
2. How to generate inputs for the predicate — using Arbitrary

The predicate block is a function that directly asserts, taking values generated by Arbitrary as input.

Arbitrary

Arbitrary generates random values. It is also responsible for shrinking those values if asked to shrink a failed value as input.

Here, we used only one type of arbitrary, Pbt.integer. There are many other built-in arbitraries, and you can create a variety of inputs by combining existing ones.

Shrink

In PBT, If a test fails, it attempts to shrink the case that caused the failure into a form that is easier for humans to understand. In other words, instead of stopping the test itself the first time it fails and reporting the failed value, it tries to find the minimal value that causes the error.

When there is a test that fails when given an even number, a counterexample of [0, -1] is simpler and easier to understand than any complex example like [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231].

Arbitrary

There are many built-in arbitraries in Pbt. You can use them to generate random values for your tests. Here are some representative arbitraries.

Primitives

Pbt.integer.generate                  # => 42
Pbt.integer(min: -1, max: 8).generate # => Integer between -1 and 8

Pbt.symbol.generate                   # => :atq

Pbt.ascii_char.generate               # => "a"
Pbt.ascii_string.generate             # => "aagjZfao"

Pbt.boolean.generate                  # => true or false
Pbt.constant(42).generate             # => 42 always

You can also pass a custom random number generator if needed:

rng = Random.new(42) # with a specific seed for reproducibility
Pbt.integer.generate(rng)

Composites

Pbt.array(Pbt.integer).generate                        # => [121, -13141, 9825]
Pbt.array(Pbt.integer, max: 1, empty: true).generate   # => [] or [42] etc.

Pbt.tuple(Pbt.symbol, Pbt.integer).generate            # => [:atq, 42]

Pbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate # => {x: :atq, y: 42}
Pbt.hash(Pbt.symbol, Pbt.integer).generate             # => {atq: 121, ygab: -1142}

Pbt.one_of(:a, 1, 0.1).generate                        # => :a or 1 or 0.1

See ArbitraryMethods module for more details.

Stateful Testing (Experimental)

Pbt also provides an experimental stateful property API for model-based / command-based testing. It is designed as a property-compatible object (generate, shrink, run) so it works with the existing runner (Pbt.assert / Pbt.check) without changing the runner API.

This API is still experimental. Expect interface refinements and behavior changes in future releases.

Minimal usage

class CounterModel
  def initialize
    @inc = IncrementCommand.new
  end

  def initial_state
    0
  end

  def commands(_state)
    [@inc]
  end
end

class IncrementCommand
  def name
    :increment
  end

  def arguments(_state)
    Pbt.nil
  end

  def applicable?(_state, _args)
    true
  end

  def next_state(state, _args)
    state + 1
  end

  def run!(sut, _args)
    sut.increment
  end

  def verify!(before_state:, after_state:, args: _, result:, sut:)
    raise "unexpected result" unless result == after_state
    raise "state mismatch" unless after_state == before_state + 1
    raise "sut mismatch" unless sut.value == after_state
  end
end

class Counter
  attr_reader :value

  def initialize
    @value = 0
  end

  def increment
    @value += 1
  end
end

Pbt.assert do
  Pbt.stateful(
    model: CounterModel.new,
    sut: -> { Counter.new },
    max_steps: 20
  )
end

Expected interfaces

Pbt.stateful(model:, sut:, max_steps:) expects the following duck-typed interfaces:

• model.initial_state
• model.commands(state) -> Array<command>
• command.name
• command.arguments(state) (a Pbt arbitrary, may depend on current model state)
• command.applicable?(state, args) -> true / false
• command.next_state(state, args) -> next model state
• command.run!(sut, args) -> command result
• command.verify!(before_state:, after_state:, args:, result:, sut:)

Current limitations (MVP)

• Pbt.stateful runs sequentially by default, even if the global worker is :ractor.
• You can still pass worker: :none explicitly if you want to make that choice obvious in a test.
• worker: :ractor is currently unsupported and raises Pbt::InvalidConfiguration.
• Shrinking supports shorter prefixes and command-argument shrinking (using command.arguments.shrink(args)).

What if property-based tests fail?

Once a test fails it's time to debug. Pbt provides some features to help you debug.

How to reproduce

When a test fails, you'll see a message like below.

Pbt::PropertyFailure:
  Property failed after 23 test(s)
  seed: 43738985293126714007411539287084402325
  counterexample: [0, -1]
  Shrunk 40 time(s)
  Got RuntimeError: Sort algorithm is wrong.
  # and backtraces

You can reproduce the failure by passing the seed to Pbt.assert.

Pbt.assert(seed: 43738985293126714007411539287084402325) do
  Pbt.property(Pbt.array(Pbt.integer)) do |number|
    # your test
  end
end

Verbose mode

You may want to know which values pass and which values fail. You can enable verbose mode by passing verbose: true to Pbt.assert.

Pbt.assert(verbose: true) do
  Pbt.property(Pbt.array(Pbt.integer)) do |numbers|
    # your failed test
  end
end

The verbose mode prints the results of each tested values.

Encountered failures were:
- [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]
- [310864, 856411, -304517, 86613, -78231]
- [-304517, 86613, -78231]
(snipped for README)
- [0, -3]
- [0, -2]
- [0, -1]

Execution summary:
. × [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]
. . √ [-897860, -930517, 577