Lotte

Lotte is a headless, automated testing framework built on top of PhantomJS and inspired by Ghostbuster. It adds jQuery-like methods and chaining, more assertion logic and an extensible core. Tests can be written in either JavaScript or CoffeeScript.

Lotte comes with tools for accessing the DOM, evaluating arbitrary code, simulating mouse and keyboard input.

Tests are sandboxed and run asynchronously. Blocking methods are available to simulate dependencies and to control the flow of execution.

This project is still highly experimental. Using it may cause your computer to blow up. Seriously.

Prerequisites

• node.js >=0.8

• npm >=1.0

• PhantomJS <2.0.0

Optional Dependencies

• CoffeeScript

Installation

$ npm -g install lotte

-global is preferred so you can run lotte from any directory.

Usage

Create a new file lotte_github.js (preferably in an empty directory) and copy the following code:

this.open('https://github.com', function() {
  this.describe('Sign Up button', function() {
    this.assert.ok(this.$('.signup-button').length, 'expects button to be in the DOM');
    this.success();
  });
});

Run lotte from within the directory and you should see the following output:

/tmp/lotte_github.js
  @ https://github.com
      ✓ Sign Up button

Command-Line Options

You can customise many aspects of Lotte's behaviour either on the command-line on through Lottefiles. The following options are available:

$ lotte --help
Usage: lotte [OPTION...] PATH

Options:
  --help, -h        give this help page
  --version, -v     print program version
  --concurrent, -c  limit of files to run in parallel                       [default: 4]
  --timeout, -t     timeout for individual files (in milliseconds)          [default: 30000]
  --include, -I     glob pattern to match files in PATH          [string]   [default: "**/lotte_*.js"]
  --exclude, -E     glob pattern to remove included files        [string]
  --lottefile, -f   look for 'lottefile' in PATH                 [string]   [default: "Lottefile"]
  --verify          verify PhantomJS version (expected <2.0.0)   [boolean]  [default: true]
  --phantom         executable for PhantomJS                     [string]
  --coffee          executable for CofeeScript                   [string]

There are four key options you would want to customise while the rest should work with their defaults.

• --concurrent, -c

  If you have more than one test file in a directory, Lotte will attempt to run them in parallel (asynchronously). You can specify how many tests can be running at any given time through this option.

  If you want to run tests synchronously, specify a value of 1.

• --timeout, -t

  Each test is expected to finish within a given period of time. If a test takes longer, it is interruped and recorded as failed.

  The default value is 30 seconds, but you should consider reducing it.

• --include, -I
--exclude, -E

  When you run lotte from any directory the script collects a list of all files in the current directory and all sub-directories. The list is reduced by running the include glob pattern and dropping any files that did not match. The list is then reduced further by running the exclude glob pattern and dropping any files that did match. The remaining list is sorted and considered final.

  You can specify these arguments more than once to create an array of include/exclude patterns.

Lottefile

In order to avoid having to type the full lotte command-line each time, you can use Lottefiles to store your settings per project.

Lottefiles are regular JavaScript files where each global variable maps to a command-line option. For example, the following command:

$ lotte --include '**/*.coffee' --include '**/*.js' --concurrent 1 tests

can be stored in a Lottefile as this:

path       = 'tests'
include    = ['**/*.coffee', '**/*.js']
concurrent = 1

Running lotte from the project directory will then read the Lottefile and scan the tests directory for all files matching **/*.{coffee,js}.

Writing Tests

Tests can be written in either JavaScript or CoffeeScript. In the sections below substitute @ with this. if you are using JavaScript. Arguments wrapped in square brackets [ and ] are optional. Arguments ending in ... can be used more than once.

At the top-level, the following functions are available:

• @title([name])

  Gets or sets the test title. This is useful for giving meaningful names to your tests.

  When called with zero arguments, returns the current title or undefined.
  When called with one argument, sets the title.

  If you don't explicitly specify a title, the filename will be used instead.

• @base([uri])

  Gets or sets the absolute URI for all relative URIs in the test. You can use this to specify the root URI for your project.

  When called with zero arguments, returns the current URI or undefined.
  When called with one argument, sets the URI.

  If you don't explicitly specify an absolute URI, all calls to @open will expect an absolute URI instead.

• @open(uri, [message], [options], block)

  Creates a new test.

  uri can be either an absolute or relative URI (see above).
  message is an optional description for the URI. If you don't specify it, Lotte will print the uri in the output instead.
  options is an object hash to pass to PhantomJS. See settings (object).
  block is a function which is executed if the server returns a valid response (2xx or 3xx).

  If the server returns a 4xx or 5xx HTTP code instead, the test is recorded as failed.

  If you have more than one @open call at the top-level, they will be executed asynchronously.

Putting it all together, a test file could look like this:

@title 'Github'
@base  'https://github.com'

@open '/', 'the homepage', ->
  # ...body of test...

Cases & Grouping

Once you have successfully requested an URI, you can start writing test cases against the page.

The following functions are available:

• @group(name, block)

  Groups the nested test cases. This is mainly for structuring the output Lotte prints.

  name is the name of the group.
  block is a function which contains the nested test cases.

• @describe(name, block)

  Starts a new test case.

  name is the name of the test case.
  block is a function which is executed and is expected to contain assertion logic.

Putting it all together a test file could now look like this:

@title 'Github'
@base  'https://github.com'

@open '/', 'the homepage', ->
  @describe 'counter shows number of repositories', ->
    # ...assertion logic...
  @group 'Sign Up button', ->
    @describe 'is in place', ->
      # ...assertion logic...
    @describe 'takes you to /plans', ->
      # ...assertion logic...

Flow of Execution

Each test case is executed in the order in which it is defined:

@describe 'I run first', ->
@describe 'I run second', ->
# etc.

If a test case contains an asynchronous function call, the next test case is executed without waiting for the function to finish:

@describe 'I run first', ->
@describe 'I run second', ->
  setTimeout ( -> ), 2500
@describe 'I run third in parallel with second still running', ->

Be extremely careful when dealing with asynchronous function. For example, using .click() to follow an anchor could change the page while another test case is running.

If a test case fails, any remaining test cases are skipped:

@describe 'I run first', ->
  throw 'Whoops!'
@describe 'I should run second, but I never will', ->

To simulate dependencies and control the flow of execution, you can use the following functions:

• @wait(name..., block)

  Blocks the current test case until all dependencies have finished (either passed or failed).

The earlier example can now be rewritten as follows to make it synchronous again:

@describe 'I run first', ->
@describe 'I run second', ->
  setTimeout ( -> ), 2500
@describe 'I run third', ->
  @wait 'I run second', ->
    # ...assertion logic...

Environments

Lotte uses PhantomJS to execute tests. While you may be writing tests in JavaScript and expect to be able to access the DOM of a page directly, this is not the case.

Each test file runs in its own sandboxed environment. Each page you request also runs in a sandbox. You cannot access variables across environments, i.e., you cannot define a variable in your test file and access it within the page you have just requested:

@open 'https://github.com', ->
  @describe 'Sandbox', ->
    val = 'value'
    # following line throws an exception
    console.log(@page.evaluate( -> return val))
    throw 'exit'

In the above code snippet @page.evaluate runs the function as if it were defined on the page you just requested, i.e, github.com. In order to do so, PhantomJS serializes the function, but it does not include the context in which it was defined. When the function is executed, val is missing in the new context causing it to throw an exception.

Another limit