quickcheck

QuickCheck is a way to do property based testing using randomly generated input. This crate comes with the ability to randomly generate and shrink integers, floats, tuples, booleans, lists, strings, options and results. All QuickCheck needs is a property function—it will then randomly generate inputs to that function and call the property for each set of inputs. If the property fails (whether by a runtime error like index out-of-bounds or by not satisfying your property), the inputs are "shrunk" to find a smaller counter-example.

The shrinking strategies for lists and numbers use a binary search to cover the input space quickly. (It should be the same strategy used in Koen Claessen's QuickCheck for Haskell.)

Dual-licensed under MIT or the UNLICENSE.

Documentation

The API is fully documented: https://docs.rs/quickcheck.

Simple example

Here's an example that tests a function that reverses a vector:

fn reverse<T: Clone>(xs: &[T]) -> Vec<T> {
    let mut rev = vec![];
    for x in xs.iter() {
        rev.insert(0, x.clone())
    }
    rev
}

#[cfg(test)]
mod tests {
    use quickcheck::quickcheck;
    use super::reverse;

    quickcheck! {
        fn prop(xs: Vec<u32>) -> bool {
            xs == reverse(&reverse(&xs))
        }
  }
}

This example uses the quickcheck! macro, which is backwards compatible with old versions of Rust.

The #[quickcheck] attribute

To make it easier to write QuickCheck tests, the #[quickcheck] attribute will convert a property function into a #[test] function.

To use the #[quickcheck] attribute, you must import the quickcheck macro from the quickcheck_macros crate:

fn reverse<T: Clone>(xs: &[T]) -> Vec<T> {
    let mut rev = vec![];
    for x in xs {
        rev.insert(0, x.clone())
    }
    rev
}

#[cfg(test)]
mod tests {
    use quickcheck_macros::quickcheck;
    use super::reverse;

    #[quickcheck]
    fn double_reversal_is_identity(xs: Vec<isize>) -> bool {
        xs == reverse(&reverse(&xs))
    }
}

Installation

quickcheck is on crates.io, so you can include it in your project like so:

[dependencies]
quickcheck = "1"

If you're only using quickcheck in your test code, then you can add it as a development dependency instead:

[dev-dependencies]
quickcheck = "1"

If you want to use the #[quickcheck] attribute, then add quickcheck_macros

[dev-dependencies]
quickcheck = "1"
quickcheck_macros = "1"

N.B. When using quickcheck (either directly or via the attributes), RUST_LOG=quickcheck enables info! so that it shows useful output (like the number of tests passed). This is not needed to show witnesses for failures.

Crate features:

• "use_logging": (Enabled by default.) Enables the log messages governed RUST_LOG.
• "regex": (Enabled by default.) Enables the use of regexes with env_logger.

Minimum Rust version policy

This crate's minimum supported rustc version is 1.85.0.

The current policy is that the minimum Rust version required to use this crate can be increased in minor version updates. For example, if crate 1.0 requires Rust 1.20.0, then crate 1.0.z for all values of z will also require Rust 1.20.0 or newer. However, crate 1.y for y > 0 may require a newer minimum version of Rust.

In general, this crate will be conservative with respect to the minimum supported version of Rust.

With all of that said, currently, rand is a public dependency of quickcheck. Therefore, the MSRV policy above only applies when it is more aggressive than rand's MSRV policy. Otherwise, quickcheck will defer to rand's MSRV policy.

Compatibility

In general, this crate considers the Arbitrary implementations provided as implementation details. Strategies may or may not change over time, which may cause new test failures, presumably due to the discovery of new bugs due to a new kind of witness being generated. These sorts of changes may happen in semver compatible releases.

Alternative Rust crates for property testing

The proptest crate is inspired by the Hypothesis framework for Python. You can read a comparison between proptest and quickcheck here and here. In particular, proptest improves on the concept of shrinking. So if you've ever had problems/frustration with shrinking in quickcheck, then proptest might be worth a try!

Alternatives for fuzzing

Please see the Rust Fuzz Book and the arbitrary crate.

Discarding test results (or, properties are polymorphic!)

Sometimes you want to test a property that only holds for a subset of the possible inputs, so that when your property is given an input that is outside of that subset, you'd discard it. In particular, the property should neither pass nor fail on inputs outside of the subset you want to test. But properties return boolean values—which either indicate pass or fail.

To fix this, we need to take a step back and look at the type of the quickcheck function:

pub fn quickcheck<A: Testable>(f: A) {
    // elided
}

So quickcheck can test any value with a type that satisfies the Testable trait. Great, so what is this Testable business?

pub trait Testable {
    fn result(&self, &mut Gen) -> TestResult;
}

This trait states that a type is testable if it can produce a TestResult given a source of randomness. (A TestResult stores information about the results of a test, like whether it passed, failed or has been discarded.)

Sure enough, bool satisfies the Testable trait:

impl Testable for bool {
    fn result(&self, _: &mut Gen) -> TestResult {
        TestResult::from_bool(*self)
    }
}

But in the example, we gave a function to quickcheck. Yes, functions can satisfy Testable too!

impl<A: Arbitrary + Debug, B: Testable> Testable for fn(A) -> B {
    fn result(&self, g: &mut Gen) -> TestResult {
        // elided
    }
}

Which says that a function satisfies Testable if and only if it has a single parameter type (whose values can be randomly generated and shrunk) and returns any type (that also satisfies Testable). So a function with type fn(usize) -> bool satisfies Testable since usize satisfies Arbitrary and bool satisfies Testable.

So to discard a test, we need to return something other than bool. What if we just returned a TestResult directly? That should work, but we'll need to make sure TestResult satisfies Testable:

impl Testable for TestResult {
    fn result(&self, _: &mut Gen) -> TestResult { self.clone() }
}

Now we can test functions that return a TestResult directly.

As an example, let's test our reverse function to make sure that the reverse of a vector of length 1 is equal to the vector itself.

fn prop(xs: Vec<isize>) -> TestResult {
    if xs.len() != 1 {
        return TestResult::discard()
    }
    TestResult::from_bool(xs == reverse(&xs))
}
quickcheck(prop as fn(Vec<isize>) -> TestResult);

(A full working program for this example is in examples/reverse_single.rs.)

So now our property returns a TestResult, which allows us to encode a bit more information. There are a few more convenience functions defined for the TestResult type. For example, we can't just return a bool, so we convert a bool value to a TestResult.

(The ability to discard tests allows you to get similar functionality as Haskell's ==> combinator.)

N.B. Since discarding a test means it neither passes nor fails, quickcheck will try to replace the discarded test with a fresh one. However, if your condition is seldom met, it's possible that quickcheck will have to settle for running fewer tests than usual. By default, if quickcheck can't find 100 valid tests after trying 10,000 times, then it will give up. These parameters may be changed using QuickCheck::tests and QuickCheck::max_tests, or by setting the QUICKCHECK_TESTS and QUICKCHECK_MAX_TESTS environment variables. There is also QUICKCHECK_MIN_TESTS_PASSED which sets the minimum number of valid tests that need pass (defaults to 0) in order for it to be considered a success.

Shrinking

Shrinking is a crucial part of QuickCheck that simplifies counter-examples for your properties automatically. For example, if you erroneously defined a function for reversing vectors as: (my apologies for the contrived example)

fn reverse<T: Clone>(xs: &[T]) -> Vec<T> {
    let mut rev = vec![];
    for i in 1..xs.len() {
        rev.insert(0, xs[i].clone())
    }
    rev
}

And a property to test that xs == reverse(reverse(xs)):

fn prop(xs: Vec<isize>) -> bool {
    xs == reverse(&reverse(&xs))
}
quickcheck(prop as fn(Vec<isize>) -> bool);

Then without shrinking, you might get a counter-example like:

[quickcheck] TEST FAILED. Arguments: ([-17, 13, -12, 17, -8, -10, 15, -19,
-19, -9, 11, -5, 1, 19, -16, 6])

Which is pretty mysterious. But with shr