Kombucha

Smart API snapshot testing and linting

Overview

Kombucha is a command-line program for performing API snapshot testing. The first time Kombucha runs, it executes the HTTP requests you specify and stores the corresponding JSON responses for those requests. These initial responses become the known-good “snapshots” of the API endpoints you are interested in.

On subsequent runs, Kombucha compares the live response for each endpoint to its corresponding snapshot, and flags differences that it finds problematic. Kombucha is smart enough to know that simple differences in the values inside an API response (eg. a value that changed from true to false, or from "monday" to "tuesday") do not constitute breakages of the contract for that API. Instead, Kombucha uses a simple heuristic to determine whether data differences are problematic.

To put it another way, Kombucha can flag for you when API contracts are broken as endpoints are modified or evolved. However, unlike some other tools, the user does not have to manually specify details of the API contract — Kombucha uses each endpoint’s snapshot as an implicit understanding of the contract.

For details on this computation see the structure check.

Usage

To spin up Kombucha tests for your endpoints, create a configuration file:

Configuration file

Kombucha’s behavior is governed by a declarative JSON configuration file. This section explains the contents of that file in detail, or you can jump into the repo and check out a sample configuration file directly.

The top level of the configuration file looks like this:

{
  "sharedHttpHeadersForSnaps": {
        "User-Agent": "kombucha-this-is-a-test-v-0.0001"
    },
  "snaps": [ ... ]
}

• sharedHttpHeadersForSnaps is used to set up HTTP headers that will be present in all requests. They can be overridden or setup for an individual snap by using the httpHeaders key on the snap object (more detail below).
• snaps is an array of API endpoints that you would like tested. Here’s what a single entry in the snaps array looks like:

{
  ...
  "snaps": [
    {
      "body": { ... },
      "host": "httpbin.org",
      "httpMethod": "GET",
      "httpHeaders": {
        "Accept-Language": "en-US"  
      },
      "path": "/response-headers",
      "queryItems": {
        "foo": "1",
        "bar": "2",
        "baz": "3"
      },
      "scheme": "https",
      "__snapName": "myUniqueSnapName",
      "__snapType": "__REST"
    },
  ...
  ]
}

• The snaps entry needs to contain the __snapName key to uniquely identify the request. This name will be used to create a file with the snap result on disk.
• It is possible to specify any HTTP header field with the httpHeaders key, the value of a header needs to be a string. This key is not required.
• The body key (not required) accepts arbitrary JSON and it will be used to set the body if the HTTP request.
• __snapType specifies the type of request Kombucha is going to issue. The __REST" type represents a simple HTTP request. The HTTP method most be specified with the required httpMethod key, use GET, POST, PUT, DELETE, PATCH or any other methods as the value. The remaining parameters are interpreted accordingly: queryItems are converted into key-value pairs and appended onto the path, etc.
  • The snaps entry above results in an HTTP GET to https://httpbin.org/response-headers?foo=1&bar=2&baz=3. In addition, Kombucha always sends an Accept header of application/json.

Configuration file for graphQL

Kombucha also supports GraphQL endpoints.

{
  ...
  "snaps": [
   {
      "host": "www.testHost.com",
      "path": "/graphql",
      "httpHeaders": {
        "Accept-Language": "en-US"  
      },      
      "queryContent": {
        "variables": {
          "variableName": "aValue"
        },
        "queryText": "query Model($variableName: String!) { model(variableName: $variableName) { variableName }}"
      },
      "scheme": "https",
      "__snapName": "test-graphql-query-text",
      "__snapType": "__GRAPHQL"
    },
    {
      "host": "www.testHost.com",
      "path": "/graphql",
      "queryContent": {
        "queryFile": "../Folder_Name/Model.graphql",
        "variables": {
          "variableName": "aValue"
        }
      },
      "scheme": "https",
      "__snapName": "test-graphql-url-query",
      "__snapType": "__GRAPHQL"
    }
    ...
  ]
}

In order to create a graphQL query, change the __snapType to __GRAPHQL and provide the queryContent object.

• Provide a URL to a query file by setting the queryFile key or provide the query string itself by setting the queryText key. If you provide a relative path, it will be resolved from the current working directory when running the tool.
• The variables for the query are provided by using the variables key.
• Optinaly, a graphQL opeation name for the query can be set by using the operationName key.

Customizing output

When executing a snapshot test, Kombucha converts both the snapshot JSON and the “work” (live response) JSON into data in memory. It then executes what we call the structure check to look for the API breakages and problems discussed above.

However, with the full JSON data in memory, we can perform other checks on the response as we work. It may be useful to think of this as a built-in “linter” for the response that runs alongside the core snapshotting feature that has previously been described.

The user can add a __preferences blob to their configuration file to specify exactly the mixture of checks they’d like to run:

{
  ...
  "snaps": [
    {
      "host": "httpbin.org",
      "httpMethod": "GET",
      "path": "/response-headers",
      "queryItems": {},
      "scheme": "https",
      "__preferences": {
        "errors": [
          "string-bools",
          "string-numbers",
          "structure"
        ],
        "infos": [
          "flag-new-keys"
        ],
        "warnings": [
          "array-consistency",
          "empty-arrays",
          "empty-objects"
        ]
      },
      "__snapName": "myUniqueSnapName",
      "__snapType": "__REST"
    },
  ...
  ]
}

• these __preferences specify that, in addition to structure, the checks named string-bools and string-numbers should also be run, and if they produce output, that output should be treated as errors, failing the test run.
  • In addition, the user has requested that the flag-new-keys check produce INFO level output, and that array-consistency, empty-arrays, and empty-objects be checked, and routed to WARN level output.
• any check name (eg. flag-new-keys) can be placed at any key (errors, infos, or warnings) in the __preferences blob.
• if no __preferences are specified, Kombucha runs the default checks. See here for the full list of available checks.

Example output

testing: GET: example.org/example - (0 query params)
wrote to: /Users/ptomaselli/Documents/Code/kombucha/__Work__/example.json
['json-path']
    > ERROR: The key new-key does not exist

testing: GET: example.org/example2 - (0 query params)
wrote to: /Users/ptomaselli/Documents/Code/kombucha/__Work__/example2.json
['json-path-2']
    > ERROR: Types didn’t match. Reference: bool(false), test: string("hello world")

testing: GET: example.org/example3 - (3 query params)
wrote to: /Users/ptomaselli/Documents/Code/kombucha/__Work__/example3.json

    > INFO: The key foo exists in the value being tested, but not in the snapshot. Perhaps you need to update your snapshot?

testing: GET: example.org/example4 - (0 query params)
wrote to: /Users/ptomaselli/Documents/Code/kombucha/__Work__/example4.json
finished with errors

Command line parameters

• [POSITIONAL] configurationURLString: The path to the json configuration file. If not specified, the default is ./kombucha.json.
• --print-errors-only (-e): ask Kombucha to omit INFO and WARNING output when printing to the console. Useful if a set of tests is producing a lot of output and you’d like to “zero in” on the errors only, without having to edit the configuration file
• --record (-r): ask Kombucha to rewrite all the stored snapshots it knows about. Useful for setting a new baseline for tests or if you know a large amount of the API under test has changed
• --snapshots-directory (-s): tell Kombucha where to find snapshots for the test run. If not specified, the default is ./__Snapshots/.
• --work-directory (-w): tell Kombucha what directory to use as the “work directory” (storage for live responses) for the test run. If not specified, the default is ./__Work__/.
• --report-type (-t): If specified, a report will be created on disk for the current test run. junit is currently the only support type. You need to specify the --report-output-url path for the report.
• --report-output-url (-o): The path to the report file (existing or not).

Running on macOS

Kombucha is written in Swift. If you have a Swift 5 development environment available, you can run Kombucha natively on macOS. To do so, clone this repository, and then run

swift build                         # build Kombucha in debug mode
.build/debug/kombucha sample.json   # run the sample tests that are included with this repo

To move forward from this point, construct your own JSON configuration file describing your API endpoints and point Kombucha at that file. We suggest storing this configuration file and its snapshots in a git repository separate from this one, so that you can version your usage of Kombucha separately from any ver