🐍 Venom

Venom is a CLI (Command Line Interface) that aims to create, manage and run your integration tests with efficiency.

<a href="https://github.com/ovh/venom/releases/latest"><img alt="GitHub release" src="https://img.shields.io/github/v/release/ovh/venom.svg?logo=github&style=flat-square"></a> <a href="https://gitpod.io/#https://github.com/ovh/venom"><img src="https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod" alt="Contribute with Gitpod"/></a>

Table of content

• 🐍 Venom
• Table of content
• Overview
• Installing
  • Install from binaries
• Updating
• Docker image
• CLI Usage
  • Run test suites in a specific order
  • Globstar support
  • Variables
    • Variable Definitions Files
    • Environment Variables
  • Arguments
    • Define arguments with environment variables
  • Use a configuration file
• Concepts
  • TestSuites
  • Executors
    • User defined executors
  • Variables
    • Testsuite variables
    • Variable helpers
  • Use outputs from a test step as input of another test step
  • Builtin venom variables
    • Secrets variables
  • Assertions
    • Keywords
      • Must keywords
      • User assertions
    • Using logical operators
• Write and run your first test suite
• Export tests report
• Advanced usage
  • Debug your testsuites
  • Skip testcase and teststeps
  • Iterating over data
• FAQ
  • Common errors with quotes
• Use venom in CI/CD pipelines
• Hacking
  • How to compile?
  • How to test?
    • Unit tests:
    • Integration tests:
• Contributing
• License

Overview

Venom allows you to handle integration tests the same way you code your application. With Venom, testcases will be managed as code: the readability of the tests means that the tests are part of the code reviews. Thanks to that, write and execute testsuites become easier for developers and teams.

Concretely, you have to write testsuite in a YAML file. Venom run executors (scripts, HTTP Request, web, IMAP, etc.) and apply assertions. It can also generate xUnit result files.

<img src="./venom.gif" alt="Venom Demonstration">

Installing

Install from binaries

You can find latest binary release from: https://github.com/ovh/venom/releases/latest/.

Example for Linux:

$ curl https://github.com/ovh/venom/releases/download/v1.2.0/venom.linux-amd64 -L -o /usr/local/bin/venom && chmod +x /usr/local/bin/venom
$ venom -h

Updating

You can update to the latest version with venom update command:

$ venom update

The venom update command will download the latest version and replace the current binary:

Url to update venom: https://github.com/ovh/venom/releases/download/v1.2.0/venom.darwin-amd64
Getting latest release from: https://github.com/ovh/venom/releases/download/v1.2.0/venom.darwin-amd64 ...
Update done.

Check the new version with venom version command:

$ venom version
Version venom: v1.2.0 

Docker image

Instead of installing (and updating) Venom locally, Venom can be started as a Docker image with following commands.

Considering your testsuites are in ./tests directory in your current directory and your test library is under ./tests/lib, the results will be available under the results directory.

$ mkdir -p results
$ docker run --mount type=bind,source=$(pwd)/tests,target=/workdir/tests --mount type=bind,source=$(pwd)/results,target=/workdir/results ovhcom/venom:latest 

Please refer to https://hub.docker.com/r/ovhcom/venom/tags to get the available image tags.

CLI Usage

venom CLI is composed of several commands:

$ venom -h
Venom - RUN Integration Tests

Usage:
  venom [command]

Available Commands:
  help        Help about any command
  run         Run Tests
  update      Update venom to the latest release version: venom update
  version     Display Version of venom: venom version

Flags:
  -h, --help   help for venom

Use "venom [command] --help" for more information about a command.

You can see the help of a command with venom [command] -h:

$ venom run -h

run integration tests

Usage:
  venom run [flags]

Examples:
  Run all testsuites containing in files ending with *.yml or *.yaml: venom run
  Run a single testsuite: venom run mytestfile.yml
  Run a single testsuite and export the result in JSON format in test/ folder: venom run mytestfile.yml --format=json --output-dir=test
  Run a single testsuite and export the result in XML and HTML formats in test/ folder: venom run mytestfile.yml --format=xml --output-dir=test --html-report
  Run a single testsuite and specify a variable: venom run mytestfile.yml --var="foo=bar"
  Run a single testsuite and load all variables from a file: venom run mytestfile.yml --var-from-file variables.yaml
  Run all testsuites containing in files ending with *.yml or *.yaml with verbosity: VENOM_VERBOSE=2 venom run

  Notice that variables initialized with -var-from-file argument can be overrided with -var argument

  More info: https://github.com/ovh/venom

Flags:
      --format string           --format:json, tap, xml, yaml (default "xml")
  -h, --help                    help for run
      --html-report             Generate HTML Report
      --lib-dir string          Lib Directory: can contain user executors. example:/etc/venom/lib:$HOME/venom.d/lib
      --output-dir string       Output Directory: create tests results file inside this directory
      --stop-on-failure         Stop running Test Suite on first Test Case failure
      --var stringArray         --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file strings   --var-from-file filename.yaml --var-from-file filename2.yaml: yaml, must contains a dictionary
  -v, --verbose count           verbose. -v (INFO level in venom.log file), -vv to very verbose (DEBUG level) and -vvv to very verbose with CPU Profiling

Run test suites in a specific order

• venom run 01_foo.yml 02_foo.yml will run 01 before 02.
• venom run 02_foo.yml 01_foo.yml will run 02 before 01.

If you want to sort many testsuite files, you can use standard commands, example:

venom run `find . -type f -name "*.yml"|sort`

Globstar support

The venom CLI supports globstar:

$ venom run ./foo/b*/**/z*.yml

Variables

To specify individual variables on the command line, use the --var option when running the venom run commands:

$ venom run --var="foo=bar"
$ venom run --var='foo_list=["biz","buz"]'
$ venom run --var='foo={"biz":"bar","biz":"barr"}'

The --var option can be used many times in a single command.

Variable Definitions Files

To set a lot of variables, it is more convenient to specify their values in a variable definitions file. This file is a YAML dictionary. You have to specify that file on the command line with --var-from-file:

venom run --var-from-file variables.yaml

Environment Variables

As a fallback for the other ways of defining variables, venom tool searches the environment of its own process for environment variables named VENOM_VAR_ followed by the name of a declared variable.

$ export VENOM_VAR_foo=bar
$ venom run *.yml

You can also define the environment variable and run your testsuite in one line:

$ VENOM_VAR_foo=bar venom run *.yml

Arguments

You can define arguments on the command line using the flag name.

Flags are listed in the result of help command.

List of available flags for venom run command:

Flags:
      --format string           --format:json, tap, xml, yaml (default "xml")
  -h, --help                    help for run
      --html-report             Generate HTML Report
      --lib-dir string          Lib Directory: can contain user executors. example:/etc/venom/lib:$HOME/venom.d/lib
      --output-dir string       Output Directory: create tests results file inside this directory
      --stop-on-failure         Stop running Test Suite on first Test Case failure
      --var stringArray         --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file strings   --var-from-file filename.yaml --var-from-file filename2.yaml: yaml, must contains a dictionary
  -v, --verbose count           verbose. -vv to very verbose and -vvv to very verbose with CPU Profiling

Define arguments with environment variables

You can also define the arguments with environment variables:

# is the same as
VENOM_FORMAT=json venom run my-test-suite.yml

# is equivalent to
venom run my-test-suite.yml --format=json

Flags and their equivalent wit