Quixir: Pure Elixir Property-based Testing

Property-based testing is a technique for testing your code by considering general properties of the functions you write. Rather than using explicit values in your tests, you instead try to define the types of the values to feed it, and the properties of the results produced.

For example, given a list, you know that reversing it should produce a list with the same number of elements. You can specify this in Quixir like this:

ptest some_list: list do
  reversed = my_reverse(some_list)
  assert length(reversed) == length(some_list)
end

This says that we're going to run a property test. It will run the block with a large number of different lists, and inside the block you can refer to each list as some_list. Inside the block, we have normal ExUnit test code: we produce a reversed copy of the list, then assert its length is the same as the original.

But what list do we actually pass in? The simple answer is "lots of them." In this particular case, we'll generate a hundred lists. These will vary in length, and vary in content, but we guarantee to include at least one empty list and one list containing a single element (as these are both common boundary cases that can break code). The overall test passes if the assertion it contains is true for all these lists.

What's The Big Deal?

Property-based testing delivers two major benefits.

First, it tests things you might not have considered when writing tests manually. It can run tens or hundreds of thousands of tests, using a range of inputs, and verify that the properties you specify are honored.

Second, and more important, writing property-based tests forces you to think about the invariants in your code: what should be true no matter what I feed this function? And invariants are the cornerstone of all good design. Most likely you use them every day, but they're often implicit in what you do. Property-based testing surfaces these invariants—they will drive (and improve) the design of your code.

’nuf hype. Here are the details. But first…

Alternatives

For a different approach, see ExCheck, built on triq.

Installation

def deps do
  [
    ...
    { :quixir, "~> 0.9", only: :test },
    ...
  ]
end

Including in Tests

Quixir tests run inside regular ExUnit tests, and can take advantage of all the ExUnit features, including tagging, setup, and describe blocks.

Here's a full test file:

defmodule TestReverse do
  use ExUnit.Case
  use Quixir

  import MyList, only: [ reverse: 1 ]

  test "a reversed list has the same length as the original" do
    ptest original: list do
      reversed = reverse(original)
      assert length(reversed) == length(original)
    end
  end

  test "reversing a list twice returns the original" do
    ptest original: list do
      new_list = original |> reverse |> reverse
      assert new_list == original
    end
  end

  test "reversing a list of length 1 does nothing" do
    ptest original: list(1) do
      assert reverse(original) == original
    end
  end

  test "reversing a list of length 2 swaps the elements" do
    ptest original: list(2) do
      [ b, a ] = reverse(original)
      assert [ a, b ] == original
    end
  end

  test "reversing a list of length 3 swaps the extremes" do
    ptest original: list(3) do
      [ c, b, a ] = reverse(original)
      assert [ a, b, c ] == original
    end
  end
end

Anatomy of a Property Test

The general form of a property test is

ptest [name1: type, name2: type, …], [option,…] do
  # code including assertions
  # this code can reference the values in name1 and name2
end

As the options are generally omitted, this simplifies to

ptest name1: type, name2: type, …  do
  # code including assertions
end

Options

repeat_for: n

Number of times to run the block, using different values each time. Defaults to 100.

trace: true

Dumps the values used in each iteration of the block.

For example:

ptest [ a: int, b: int ], trace: true, repeat_for: 50 do
  assert a + b == b + a
end

Type Specifications

A type specification is the name of a Quixir type generator, optionally followed by a keyword list of constraints.

• int
• int(min: 20, max: 50)
• int(must_have: [ 0, 10, 100 ])

There's a full list of these generators, their constraints, and their defaults, below.

Sometimes type specifications can be nested. For example, this specifies (possibly empty) lists of positive integers.

• list(of: int(min: 1))

And this is a generator for keyword lists:

• list(of: tuple(like: { atom, string })

Back references to values

Occasionally you want to make the constraints of one type depend on the value generated for a prior type. You do this using the pin operator, ^. For example, the following generates sets of two integers where the second is guaranteed to be greater the first:

ptest a: int, b: int(min: ^a + 1) do
  assert a < b
end

Examples

(These examples don't show the test "xxxx" do/end wrappers.)

ptest numbers: list(choose(from: [ int, float ])) do
  # numbers will be a randomly sized list containing
  # a mixture of ints and floats
end

ptest x: positive_int(y: value(^x * ^x)) do
  # x is a random positive integer, and y is the square
  # of that integer
end

ptest x: positive_int, y: int(min: ^x+1), z: int(min: ^y+1)  do
  # x is a random positive integer, y is larger than x,
  # and z is larger than y
end

ptest options: map(of: { atom, string}, min: 3, max: 7) do
  # options will be a map with between 3 and 7 entries.
  # each entry will have an atom as a key and a string
  # as a value.
end

ptest options: map(like: %{ name: string, age: int(min:0, max: 130) }) do
  # options will be a map with two elements, a name and an age.
  # The name will be a string, and the age an integer
  # betweem 0 and 130
end

ptest options: list(of: { atom, string}, min: 3, max: 7) do
  # options will be a keyword list with between 3 and 7 entries.
end

defmodule Person do
  defstruct name: "", age: 0
end

ptest person: struct(Person) do
  # person will be instances of struct person. Because the
  # default name is a string, the name in this test struct
  # will be a random string. Similarly, age will be a random
  # integer
end

ptest person: struct(%Person{ name: string(chars: :ascii),
                              age:  int(min: 1, max: 125)) do
  # This time, the name will be a random string of 7-bit ascii,
  # and the age will be an integer from 1 to 125.
end

List of Type Generators

Quixir uses the Pollution library to create the streams of values that are injected into the tests. These generators are documented in HexDocs. Here's a (poorly formatted) version:

<!-- pollution -->

• any()

  Generates a stream of values of any of the types: atom, float, int, list, map, string, and tuple. Structs are not included, as they require additional information to create.

  If you need finer control over the types and values returned, see the choose/2 function.

• atom(options \\ [])

  Return a stream of atoms. The characters in the atom are drawn from the ASCII printable set (space through ~).

  Example:

  iex> import Pollution.{Generator, VG}
  iex> atom(max: 10) |> as_stream |> Enum.take(5)
  [:"", :"Kv0{LGp", :"?0HX"y", :ad, :"DrS=t(Q"]
  

  Options

  • min: length

    The minimum length of an atom that will be generated (default: 0).

  • max: length

    The maximum length of an atom that will be generated (default: 255).

  • must_have: [ value, … ]

    Values that must be included in the results. There are no must-have vaules by default.

• bool()

  Return a stream of random booleans (true or false).

  Example

    iex> import Pollution.{Generator, VG}
    iex> bool |> as_stream |> Enum.take(5)
    [true, false, true, true, false]
  

• choose(options)

  Each time a value is needed, randomly choose a generator from the list and invoke it.

  Example

    iex> import Pollution.{Generator, VG}
    iex> choose(from: [ int(min: 3, max: 7), bool ]) |> as_stream |> Enum.take(5)
    [6, false, 4, true, true]
  

• float(options \\ [])

  Return a stream of random floating point numbers.

  Example

    iex> import Pollution.{Generator, VG}
    iex> float |> as_stream |> Enum.take(5)
    [0.0, -1.0, 1.0, 5.0e-324, -5.0e-324]
  

  Options

  • min: value

    The minimum value that will be generated (default: -1e6).

  • max: value

    The maximum value that will be generated (default: 1e6).

  • must_have: [ value, … ]

    Values that must be included in the results. The default is

    [ 0.0, -1.0, 1.0, epsilon, -epsilon ]

    (where epsilon is the smallest expressible float)

    Must have values are automatically adjusted to account for the min and max values. For example, if you specify min: 0.5 then only the 1.0 must-have value will be generated.

  See also

  • positive_float() • negative_float • nonnegative_float

• int(options \\ [])

  Return a stream of random integers.

  Example

    iex> import Pollution.{Generator, VG}
    iex> int |> as_stream |> Enum.take(5)
    [0, -1, 1, 215, -401]
  

  Options

  • min: value

    The minimum value that will be generated (default: -1000).

  • max: _val