judge

Judge is a library for writing inline snapshot tests in Janet. You can install it with jpm:

# project.janet
(declare-project
  :dependencies [
    {:url "https://github.com/ianthehenry/judge.git"
     :tag "v2.11.0"}
  ])

Judge tests work a little differently than traditional tests. Instead of assertions, you write expressions to observe. Like this:

(test (+ 1 1))

When you run Judge, it will replace the source code with the result of this expression:

(test (+ 1 1) 2)

The Judge test runner gives you a lot flexibility over how you structure your tests. You can put all your tests in a test/ subdirectory, following standard Janet convention, or you can put tests right next to the code that you're testing:

# sort.janet
(use judge)

(defn slow-sort [list]
  (case (length list)
    0 list
    1 list
    2 (let [[x y] list] [(min x y) (max x y)])
    (do
      (def pivot (in list (math/floor (/ (length list) 2))))
      (def bigs (filter |(> $ pivot) list))
      (def smalls (filter |(< $ pivot) list))
      [;(slow-sort smalls) pivot ;(slow-sort bigs)])))

(test (slow-sort [3 1 4 2]))

Run your tests with the Judge test runner:

$ judge
# sort.janet

- (test (slow-sort [3 1 4 2]))
+ (test (slow-sort [3 1 4 2]) [1 2 3 4])

0 passed 1 failed

And look! It fixed your tests:

# sort.janet.tested
(use judge)

(defn slow-sort [list]
  (case (length list)
    0 list
    1 list
    2 (let [[x y] list] [(min x y) (max x y)])
    (do
      (def pivot (in list (math/floor (/ (length list) 2))))
      (def bigs (filter |(> $ pivot) list))
      (def smalls (filter |(< $ pivot) list))
      [;(slow-sort smalls) pivot ;(slow-sort bigs)])))

(test (slow-sort [3 1 4 2]) [1 2 3 4])

You can then diff the .tested file with your original source and interactively merge them using whatever tools you are comfortable with.

Judge supports "anonymous" tests, as seen above, and named tests, which can group multiple (test) invocations together:

(deftest "sorting tests"
  (test (slow-sort [3 1 2 4]) [1 2 3 4])
  (test (slow-sort [1 1 1 1]) [1 1 1 1]))

When you aren't using the judge test runner, all of the macros exposed by Judge are no-ops. So these tests will never execute during normal evaluation: tests won't slow down your program, and you can freely distribute modules with Judge tests as libraries without your users even knowing.

Usage

Judge distributes a runner executable called judge. When you install Judge using jpm deps -l, the runner script will live at jpm_tree/bin/judge. You can invoke it directly as jpm_tree/bin/judge, or you can add the local bin directory to your PATH:

export PATH="./jpm_tree/bin:$PATH"

So that you can just run it as judge.

$ judge --help
Test runner for Judge.

  judge [FILE[:LINE:COL]]...

If no targets are given on the command line, Judge will look for tests in the
current working directory.

Targets can be file names, directory names, or FILE:LINE:COL to run a test at a
specific location (which is mostly useful for editor tooling).

=== flags ===

  [--help]                   : Print this help text and exit
  [-a], [--accept]           : overwrite all source files with .tested files
  [--not FILE[:LINE:COL]]... : skip all tests in this target
  [-i], [--interactive]      : select which replacements to include
  [--not-name-exact NAME]... : skip tests whose name is exactly this prefix
  [--name-exact NAME]...     : only run tests with this exact name
  [--not-name PREFIX]...     : skip tests whose name starts with this prefix
  [--name PREFIX]...         : only run tests whose name starts with the given
                               prefix
  [--color], [--no-color]    : default is --color unless the NO_COLOR environment
                               variable is set
  [-u], [--untrusting]       : re-evaluate all trust expressions
  [-v], [--verbose]          : verbose output

Writing tests

test

(test (+ 1 2) 3)

test-error

Requires that the provided expression raises an error:

(test-error (in [1 2 3] 5) "expected integer key for tuple in range [0, 3), got 5")

test-stdout

(test-stdout (print "hello") `
  hello
`)

If the expression to test does not evaluate to nil, it will be included in the test as well:

(defn add [a b]
  (printf "adding %q and %q" a b)
  (+ a b))

(test-stdout (add 1 2) `
  adding 1 and 2
` 3)

Due to ambiguity in the Janet parser for multi-line strings, a trailing newline will always be added to the output if it does not exist.

test-pp

The normal (test) does some debatable rewriting to make expressions reflect their input a little more cleanly. In particular, if you write a tuple:

(test [1 2 3])

It will correct like this:

(test [1 2 3] [1 2 3])

Even though the canonical way to print out such a tuple in Janet would look like this:

(test [1 2 3] (1 2 3))

This representation is also ambiguous, because the output renders bracketed and parenthesized tuples identically:

(test '[1 2 3] [1 2 3])
(test '(1 2 3) [1 2 3])

To avoid this ambiguity, the test-pp macro uses the same representation that Janet uses when you call (pp): paren tuples are printed with parens, and only bracketed tuples are printed with square brackets:

(test-pp '(1 2 3) (1 2 3))
(test-pp '[1 2 3] [1 2 3])

Apart from this different handling of tuple shapes, test-pp is the same as test.

trust

trust is like test, but the expression under test will only be evaluated if there is no expectation already. Once you accept a result, it will be re-used on all subsequent runs.

(trust (+ 1 2))

Will become:

(trust (+ 1 2) 3)

Just like test. But:

(trust (+ 1 2) 4)

Will still pass, because trust will not re-evaluate (+ 1 2) when there is already an expected value.

This is not very useful by itself, but if you save the result of the trust expression, you can use it to write deterministic tests against impure functions that you cache literally in your source code:

(def posts
  (trust (download-posts-from-the-internet)
    [{:id 4322
      :content "test post please ignore"}
     {:id 4321
      :content "is anybody here?"}]))
(test (format-posts posts)
  "1. test post please ignore\n2. is anybody here?")

Note that the result will be read as a quoted form.

To re-evaluate trust expressions, you can either delete specific expectations and re-run Judge, or run Judge with --untrusting to re-evaluate all trust expressions.

test-macro

test-macro is like testing the result of a macex1 expression, but the output is pretty-printed according to Janet code formatting conventions:

(test-macro (let [x 1] x)
  (do
    (def x 1)
    x))

And test-macro will replace gensym'd identifiers with stable symbols:

(test-macro (and x (+ 1 2))
  (if (def <1> x)
    (+ 1 2)
    <1>))

test-macro tries to format its output nicely, but if you've defined custom macros that you include in the expansion of the macro that you're testing, Judge won't know how to format them correctly. For example:

(defmacro scope [exprs] ~(do ,;exprs))

(defmacro twice [expr]
  ~(scope
    ,expr
    ,expr))

(test-macro (twice (print "hello")))

Will produce the rather ugly:

(test-macro (twice (print "hello"))
  (scope (print "hello") (print "hello")))

You can fix this by applying metadata to your macro binding that tells Judge how to format it. Let's say that scope should format like a block by adding the fmt/block metadata:

(defmacro scope :fmt/block [exprs] ~(do ,;exprs))

(defmacro twice [expr]
  ~(scope
    ,expr
    ,expr))

(test-macro (twice (print "hello")))

That will produce the much nicer looking:

(test-macro (twice (print "hello"))
  (scope
    (print "hello")
    (print "hello")))

There are only two format specifiers: fmt/block and fmt/control. A "block" macro formats like do: the macro name is on a line of its own. A "control" macro formats like while: the first argument is on its own line, and all subsequent arguments are on their own lines.

deftest

The first form passed to the (deftest) macro is the name of the test. It can be a symbol or a string:

(use judge)

(deftest math
  (test (+ 2 2) 4))

(deftest "advanced math"
  (test (* 2 2) 4))

You don't have to use deftest, though. You can create anonymous, single-expression tests by using any of the test macros at the top level:

(use judge)

(test (+ 1 2) 3)

Custom testing macros

You can write macros that wrap any of the existing test-macros using defmacro*. For example:

(defmacro* test-loudly [exp & args]
  ~(test (string/ascii-upper ,exp) ,;args))

(test-loudly "hi" "HI")

The only difference between defmacro and defmacro* is that defmacro* copies the source map from the macro to its expansion, which Judge needs in order to patch code.

Running tests

Run all tests in a particular file:

$ judge tests.janet

Or a directory:

$ judge tests/

Run a specific named test:

$ judge --name 'two plus'

Run test on a specific line/column (useful for editor tooling):

$ judge test.janet:10:2

Context-dependent tests

Sometimes you might have a bunch of tests that all need some kind of shared context -- a SQL connection, maybe, or an OpenGL graphics context. You could create that context anew at the beginning of every test, but that might be very expensive. There are some cases where it might be appropriate to create the context a single time, and pass it in to every test of that type.

To declare a new context-dependent test type, use the deftest-type macro:

(deftest-type stateful