Depicted—dpxdt

Make continuous deployment safe by comparing before and after webpage screenshots for each release. Depicted shows when any visual, perceptual differences are found. This is the ultimate, automated end-to-end test.

View the test instance here

Depicted is:

A local command-line tool for doing perceptual diff testing.
An API server and workflow for capturing webpage screenshots and automatically generating visual, perceptual difference images.
- A workflow for teams to coordinate new releases using pdiffs.
- A client library for integrating the server with existing continuous integration.
- Built for portability; server runs with SQLite, MySQL, behind the firewall, etc.
A wrapper of PhantomJS for screenshots.
Open source, Apache 2.0 licensed.
Not a framework, not a religion.

Depicted is not finished! Please let us know if you have feedback or find bugs.

See this video for a presentation about how perceptual diffs have made continuous deployment safe.

Local Depicted

Local dpxdt is a tool for generating screenshots. It reads in a YAML config file and generates screenshots using PhantomJS. This makes sense for testing reusable tools (e.g. libraries) which aren't "deployed" in the way that a traditional web app is.

To get started with run:

pip install dpxdt

Create a simple page to screenshot:

<!-- demo.html -->
<h2>dpxdt local demo</h2>
<p>dpxdt can be used to spot changes on local web servers.</p>

And a config file to specify what to screenshot:

# (tests/test.yaml)
# This is run once before any individual test.
# It's a good place to start your demo server.
setup: |
    python -m SimpleHTTPServer

# This runs after the setup script and before any tests are run.
# It's a great place to wait for server startup.
waitFor:
    url: http://localhost:8000/demo.html
    timeout_secs: 5

tests:
  - name: demo
    url: http://localhost:8000/demo.html
    config: {}

The setup stanza is a bash script which typically starts the server you want to screenshot. Note that this script doesn't terminate—dpxdt will kill it when it's done.

The waitFor stanza tells dpxdt how it will know whether the server is ready for screenshotting. This stanza can be a bash script as well, but typically it suffices to specify a URL and a timeout. dpxdt will repeatedly fetch the URL until it resolves (i.e. returns status code 200).

The tests section specifies a list of URLs to fetch. The name value identifies the test and is also used as the output file name.

You can run this simple test via dpxdt update tests. Here's what it looks like:

$ dpxdt update tests
Request for http://localhost:8000/demo.html succeeded, continuing with tests...
demo: Running webpage capture process
demo: Updated tests/demo.png

This looks for YAML files in the tests directory and processes each in turn. It starts the SimpleHTTPServer, captures a screenshot and writes it to tests/demo.png:

A screenshot of the demo page.

Configuration

The screenshot is 400x300 with a transparent background. This is probably not what you want! You can override these settings with the config stanza:

# (tests/test.yaml)
setup: |
    python -m SimpleHTTPServer

waitFor:
    url: http://localhost:8000/demo.html
    timeout_secs: 5

tests:
  - name: demo
    url: http://localhost:8000/demo.html
    config:
        viewportSize:
            width: 800
            height: 600
        injectCss: |
            body {
              background-color: white;
            }

You can find a complete list of config settings in capture.js, but the most common ones are viewportSize, injectCss and injectJs.

Perceptual Diffs

Local dpxdt has two modes: update and test. As we've seen above, update saves screenshots to the test directory. test takes screenshots and compares them to the saved versions.

For example:

$ dpxdt test tests
Request for http://localhost:8000/demo.html succeeded, continuing with tests...
demo: Running webpage capture process
demo: Resizing reference image
demo: Running perceptual diff process
demo passed (no diff)
All tests passed!

Now if we change demo.html:

<!-- demo.html -->
<h2>dpxdt local demo</h2>
<p>dpxdt may be used to spot changes on local web servers.</p>

and run the test command again:

$ dpxdt test tests
Request for http://localhost:8000/demo.html succeeded, continuing with tests...
demo: Running webpage capture process
demo: Resizing reference image
demo: Running perceptual diff process
demo failed
  0.0385669 distortion
  Ref:  /tmp/.../tmp6rMuVH/ref_resized
  Run:  /tmp/.../tmp6rMuVH/screenshot.png
  Diff: /tmp/.../tmp6rMuVH/diff.png
 (all):     /tmp/.../tmp6rMuVH/{ref_resized,screenshot.png,diff.png}
1 test(s) failed.

dpxdt has output a triplet of images: reference, run and diff. You can open all of them at once by copying the (all) line. For example, on Mac OS X:

open /tmp/.../tmp6rMuVH/{ref_resized,screenshot.png,diff.png}

Here's what they look like:

| Which | Image | | ---: | ----- | | Reference | reference image | | Run | run image | | Diff | diff image |

The red portions of the diff image highlight where the change is. These might be difficult to spot without a perceptual diff!

Shared Configurations

Most tests will share a similar config stanza. As your you write more tests, this gets quite repetitive. YAML supports a syntax for references which greatly reduces the repetition:

# (tests/test.yaml)
setup: |
    python -m SimpleHTTPServer

waitFor:
    url: http://localhost:8000/demo.html
    timeout_secs: 5

standard-config: &stdconfig
    viewportSize:
        width: 800
        height: 600
    injectCss: >
        body {
          background-color: white;
        }

tests:
  - name: demo
    url: http://localhost:8000/demo.html
    config: *stdconfig

  - name: demo-with-click
    url: http://localhost:8000/demo.html
    config:
        <<: *stdconfig
        injectJs: |
            $('button').click();

As the last example shows, you can "mix in" a config and add to it. If you include a stanza which is already in the mixed-in config (e.g. viewportSize), it will override it.

Depicted Server

Depicted is written in portable Python. It uses Flask and SQLAlchemy to make it easy to run in your environment. It works with SQLite out of the box right on your laptop. The API server can also run on VMs. The workers run ImageMagick and PhantomJS as subprocesses. I like to run the worker on a cloud VM, but you could run it on your behind a firewall if that's important to you.

Topics in this section:

Running the server locally
How to use Depicted effectively
Example tools
The API documentation
Deployment to production (Sqlite, etc)

Running the server locally

WARNING LABEL: This is only for local development. If you actually want a reliable server that will run for days, see the section on deployment below.

Have a version of Python 2.7 installed.
Download PhantomJS for your machine.
Download ImageMagick for your machine.

Clone this git repo in your terminal:

 git clone https://github.com/bslatkin/dpxdt.git

cd to the repo directory:
Create a new python virtual environment and activate it:
```
 virtualenv .
 source bin/activate
```

Install all dependencies into the environment:

 pip install -r requirements.txt
 pip install -e .

Execute ./run_shell.sh and run these commands to initialize your DB:
```
 server.db.drop_all()
 server.db.create_all()
```
Run the combined server/worker with ./run_combined.sh.
Navigate to http://localhost:5000.
Login and create a new build.

Execute the ./run_url_pair_diff.sh tool to verify everything is working:

 ./run_url_pair_diff.sh \
     --upload_build_id=1 \
     http://www.google.com \
     http://www.yahoo.com

Follow the URL the tool writes to the terminal and verify screenshots are present. Any errors will be printed to the log in the terminal where you are running the server process.
Deactivate your virtual environment
```
 deactivate
```

How to use Depicted effectively

Here are the steps to making Depicted useful to you:

Establish a baseline release with an initial set of screenshots of your site.
Create a new release with a new set of screenshots of your new version.
Manually approve or reject each difference the tool finds.
Manually mark the new release as good or bad.
Repeat. Your approved release will become the baseline for the next one.

Depicted organizes your releases by a build ID. You can create a build through the API server's UI. A build is usually synonymous with a binary that's pushed to production. But it could also be a unique view into one product, like one build for desktop web and another for mobile web.

Within a build are releases with names. I recomme

Dpxdt

Install / Use

README