Tasty

Tasty is a modern testing framework for Haskell.

It lets you combine your unit tests, golden tests, QuickCheck/SmallCheck properties, and any other types of tests into a single test suite.

Features:

• Run tests in parallel but report results in a deterministic order
• Filter the tests to be run using patterns specified on the command line
• Hierarchical, colored display of test results
• Reporting of test statistics
• Acquire and release resources (sockets, temporary files etc.) that can be shared among several tests
• Extensibility: add your own test providers and ingredients (runners) above and beyond those provided

To find out what's new, read the change log.

Example

Here's how your test.hs might look like:

import Test.Tasty
import Test.Tasty.SmallCheck as SC
import Test.Tasty.QuickCheck as QC
import Test.Tasty.HUnit

import Data.List
import Data.Ord

main = defaultMain tests

tests :: TestTree
tests = testGroup "Tests" [properties, unitTests]

properties :: TestTree
properties = testGroup "Properties" [scProps, qcProps]

scProps = testGroup "(checked by SmallCheck)"
  [ SC.testProperty "sort == sort . reverse" $
      \list -> sort (list :: [Int]) == sort (reverse list)
  , SC.testProperty "Fermat's little theorem" $
      \x -> ((x :: Integer)^7 - x) `mod` 7 == 0
  -- the following property does not hold
  , SC.testProperty "Fermat's last theorem" $
      \x y z n ->
        (n :: Integer) >= 3 SC.==> x^n + y^n /= (z^n :: Integer)
  ]

qcProps = testGroup "(checked by QuickCheck)"
  [ QC.testProperty "sort == sort . reverse" $
      \list -> sort (list :: [Int]) == sort (reverse list)
  , QC.testProperty "Fermat's little theorem" $
      \x -> ((x :: Integer)^7 - x) `mod` 7 == 0
  -- the following property does not hold
  , QC.testProperty "Fermat's last theorem" $
      \x y z n ->
        (n :: Integer) >= 3 QC.==> x^n + y^n /= (z^n :: Integer)
  ]

unitTests = testGroup "Unit tests"
  [ testCase "List comparison (different length)" $
      [1, 2, 3] `compare` [1,2] @?= GT

  -- the following test does not hold
  , testCase "List comparison (same length)" $
      [1, 2, 3] `compare` [1,2,2] @?= LT
  ]

And here is the output of the above program:

(Note that whether QuickCheck finds a counterexample to the third property is determined by chance.)

Packages

tasty is the core package. It contains basic definitions and APIs and a console runner.

In order to create a test suite, you also need to install one or more «providers» (see below).

Providers

The following providers exist:

• tasty-hunit — for unit tests (based on HUnit)
• tasty-golden — for golden tests, which are unit tests whose results are kept in files
• tasty-smallcheck — exhaustive property-based testing (based on smallcheck)
• tasty-quickcheck — for randomized property-based testing (based on QuickCheck)
• tasty-hedgehog — for randomized property-based testing (based on Hedgehog)
• tasty-hspec — for Hspec tests
• tasty-leancheck — for enumerative property-based testing (based on LeanCheck)
• tasty-program — run external program and test whether it terminates successfully
• tasty-wai — for testing wai endpoints
• tasty-inspection-testing — for compile-time testing of code properties (based on inspection-testing)
• tasty-flaky — add delay and retry logic to any test that is known to fail intermittently

It's easy to create custom providers using the API from Test.Tasty.Providers.

Ingredients

Ingredients represent different actions that you can perform on your test suite. One obvious ingredient that you want to include is one that runs tests and reports the progress and results.

Another standard ingredient is one that simply prints the names of all tests.

It is possible to write custom ingredients using the API from Test.Tasty.Runners.

Some ingredients that can enhance your test suite are:

• tasty-ant-xml adds a possibility to write the test results in a machine-readable XML format, which is understood by various CI systems and IDEs
• tasty-rerun adds support for minimal test reruns by recording previous test runs and using this information to filter the test tree. For example, you can use this ingredient to only run failed tests, or only run tests that threw an exception.
• tasty-html adds the possibility to write the test results as a HTML file
• tasty-stats adds the possibility to collect statistics of the test suite in a CSV file.

Test discovery

tasty by itself forces you to explicitly write out the TestTree yourself. The packages listed below allow you to write tests at the top-level, and will automatically collect them into a single TestTree.

• tasty-th
• tasty-discover
• tasty-autocollect

Other packages

• tasty-hunit-adapter converts existing HUnit test suites into tasty test suites
• tasty-expected-failure provides test markers for when you expect failures or wish to ignore tests.
• tasty-bench covers performance regression testing and extends tasty to a benchmark framework similar to criterion and gauge.

Options

Options allow one to customize the run-time behavior of the test suite, such as:

• mode of operation (run tests, list tests, run tests quietly etc.)
• which tests are run (see «Patterns» below)
• parameters of individual providers (like depth of search for SmallCheck)

Setting options

There are two main ways to set options:

Runtime

When using the standard console runner, the options can be passed on the command line or via environment variables. To see the available options, run your test suite with the --help flag. The output will look something like this (depending on which ingredients and providers the test suite uses):

% ./test --help
Mmm... tasty test suite

Usage: test [-p|--pattern PATTERN] [-t|--timeout DURATION] [--no-progress]
            [-l|--list-tests] [-j|--num-threads NUMBER] [-q|--quiet]
            [--hide-successes] [--color never|always|auto] [--ansi-tricks ARG]
            [--smallcheck-depth NUMBER] [--smallcheck-max-count NUMBER]
            [--quickcheck-tests NUMBER] [--quickcheck-replay SEED]
            [--quickcheck-show-replay] [--quickcheck-max-size NUMBER]
            [--quickcheck-max-ratio NUMBER] [--quickcheck-verbose]
            [--quickcheck-shrinks NUMBER]

Available options:
  -h,--help                Show this help text
  -p,--pattern PATTERN     Select only tests which satisfy a pattern or awk
                           expression
  -t,--timeout DURATION    Timeout for individual tests (suffixes: ms,s,m,h;
                           default: s)
  --no-progress            Do not show progress
  -l,--list-tests          Do not run the tests; just print their names
  -j,--num-threads NUMBER  Number of threads to use for tests execution
                           (default: # of cores/capabilities)
  -q,--quiet               Do not produce any output; indicate success only by
                           the exit code
  --hide-successes         Do not print tests that passed successfully
  --min-duration-to-report DURATION
                           The minimum amount of time a test can take before
                           tasty prints timing information (suffixes: ms,s,m,h;
                           default: s)
  --color never|always|auto
                           When to use colored output (default: auto)
  --ansi-tricks ARG        Enable various ANSI terminal tricks. Can be set to
                           'true' or 'false'. (default: true)
  --smallcheck-depth NUMBER
                           Depth to use for smallcheck tests
  --smallcheck-max-count NUMBER
                           Maximum smallcheck test count
  --quickcheck-tests NUMBER
                           Number of test cases for QuickCheck to generate.
                           Underscores accepted: e.g. 10_000_000
  --quickcheck-replay SEED Random seed to use for replaying a previous test run
                           (use same --quickcheck-max-size)
  --quickcheck-show-replay Show a replay token for replaying tests
  --quickcheck-max-size NUMBER
                           Size of the biggest test cases quickcheck generates
  --quickcheck-max-ratio NUMBE