Humbug: Mutation Testing for PHP

🚨 This package is deprecated, check out Infection instead.

Humbug is a Mutation Testing framework for PHP. It is currently in development and so, while it does actually work quite well, it will have rough edges that a team of minions are working hard to hammer out. If it falls out of the gate, you have been warned ;).

⚠️️ Update your remotes! Humbug has transferred to a new location. While your existing repositories will redirect transparently for any operations, take some time to transition to the new URL.

$ git remote set-url upstream https://github.com/humbug/humbug.git

Replace upstream with the name of the remote you use locally; upstream is commonly used but you may be using something else. You may also using a different URL (e.g. git@github.com:mockery/mockery.git). Run git remote -v to see what you're actually using.

Table of Contents

• Introduction
• Contributing
• Installation
  • Git
  • Phar
  • Composer
• Usage
  • Configuration
    • Configure command
    • Manual Configuration
  • Running Humbug
    • The Metrics
• Command Line Options
  • Overriding The Configured Timeout
  • Restricting Files To Mutate
  • Mutate specific files
  • Incremental Analysis
• Performance
• Mutators

Introduction

Mutation Testing is, in a nutshell, giving your unit tests a run for their money. It involves injecting small defects into source code and then checking if the unit tests noticed. If they do, then your unit tests have "killed" the mutation. If not, the mutation has escaped detection. As unit tests are intended to prevent regressions, having a real regression pass unnoticed would be a bad thing!

Whereas Code Coverage can tell you what code your tests are executing, Mutation Testing is intended to help you judge how well your unit tests actually perform and where they could be improved.

I've written in more detail about why Mutation Testing is worth having: Lies, Damned Lies and Code Coverage: Towards Mutation Testing

Contributing

Humbug is an open source project that welcomes pull requests and issues from anyone. Before opening pull requests, please read our short Contribution Guide.

Installation

Git

You can clone and install Humbug's dependencies using Composer:

git clone https://github.com/humbug/humbug.git
cd humbug
/path/to/composer.phar install

The humbug command is now at bin/humbug.

Phar

If you don't want to track the master branch directly, you can install the Humbug phar as follows:

wget https://padraic.github.io/humbug/downloads/humbug.phar
wget https://padraic.github.io/humbug/downloads/humbug.phar.pubkey
# If you wish to make humbug.phar directly executable
chmod +x humbug.phar

On Windows, you can just download using a browser or from Powershell v3 using the following commands where wget is an alias for Invoke-WebRequest:

wget https://padraic.github.io/humbug/downloads/humbug.phar -OutFile humbug.phar
wget https://padraic.github.io/humbug/downloads/humbug.phar.pubkey -OutFile humbug.phar.pubkey

If you're stuck with Powershell v2:

$client = new-object System.Net.WebClient
$client.DownloadFile("https://padraic.github.io/humbug/downloads/humbug.phar", "humbug.phar")
$client.DownloadFile("https://padraic.github.io/humbug/downloads/humbug.phar.pubkey", "humbug.phar.pubkey")

PHAR Updates

The phar is signed with an openssl private key. You will need the pubkey file to be stored beside the phar file at all times in order to use it. If you rename humbug.phar to humbug, for example, then also rename the key from humbug.phar.pubkey to humbug.pubkey.

The phar releases are currently done manually so they will not be updated with the same frequency as git master. To update your current phar, just run:

./humbug.phar self-update

Note: Using a phar means that fixes may take longer to reach your version, but there's more assurance of having a stable development version. The public key is downloaded only once. It is re-used by self-update to verify future phar releases.

Once releases commence towards stable, there will be an alpha, beta, RC and a final release. Your development track phar file will self-update automatically until it reaches a stable release. If you wish to continue tracking the development level phars, you will need to indicate this using one of the stability flags:

./humbug.phar self-update --dev

Self-Update Request Debugging

If you experience any issues self-updating with unexpected openssl or SSL errors, please ensure that you have enabled the openssl extension. On Windows, you can do this by adding or uncommenting the following line in the php.ini file for PHP on the command line (if different than the file for your http server):

extension=php_openssl.dll

Certain other SSL errors may arise due missing certificates. You can rectify this by finding their location on your system (e.g. C:/xampp/php/ext/cacert.pem), or alternatively downloading a copy from http://curl.haxx.se/ca/cacert.pem. Then ensure the following option is correctly pointing to this file:

openssl.cafile=C:/path/to/cacert.pem

Composer

Due to Humbug's dependencies being pegged to recent versions, adding Humbug to composer.json may give rise to conflicts. The above two methods of installation are preferred where this occurs. You can however install it globally as any other general purpose tool:

composer global require 'humbug/humbug=~1.0@dev'

And if you haven't done so previously...add this to ~/.bash_profile (or ~/.bashrc):

export PATH=~/.composer/vendor/bin:$PATH

Humbug currently works on PHP 5.4 or greater.

Usage

Configuration

Humbug is still under development so, to repeat, beware of rough edges.

Configure command

To configure humbug in your project you may run:

humbug configure

This tool will ask some questions required to create the Humbug configuration file (humbug.json.dist).

Manual Configuration

In the base directory of your project create a humbug.json.dist file:

{
    "timeout": 10,
    "source": {
        "directories": [
            "src"
        ]
    },
    "logs": {
        "text": "humbuglog.txt",
        "json": "humbuglog.json"
    }
}

You can commit the humbug.json.dist to your VCS and override it locally with a humbug.json file.

Edit as appropriate. If you do not define at least one log, detailed information about escaped mutants will not be available. The Text log is human readable. If source files exist in the base directory, or files in the source directories must be excluded, you can add exclude patterns (here's one for files in base directory where composer vendor and Tests directories are excluded):

{
    "timeout": 10,
    "source": {
        "directories": [
            "."
        ],
        "excludes": [
            "vendor",
            "Tests"
        ]
    },
    "logs": {
        "text": "humbuglog.txt"
    }
}

If, from your project's base directory, you must run tests from another directory then you can signal this also. You should not need to run Humbug from any directory other than your project's base directory.

{
    "chdir": "tests",
    "timeout": 10,
    "source": {
        "directories": [
            "src"
        ],
    }
}

Running Humbug

Ensure that your tests are all in a passing state (incomplete and skipped tests are allowed). Humbug will quit if any of your tests are failing.

The magic command, while in your project's base directory (using the PHAR download) is:

./humbug.phar

or if you just cloned Humbug:

../humbug/bin/humbug

or if you added Humbug as a composer dependency to your project:

./vendor/bin/humbug

Instead of php with the xdebug extension you may also run Humbug via phpdbg:

phpdbg -qrr humbug.phar

If all went well, you will get something similar to:

 _  _            _
| || |_  _ _ __ | |__ _  _ __ _
| __ | || | '  \| '_ \ || / _` |
|_||_|\_,_|_|_|_|_.__/\_,_\__, |
                          |___/ 
Humbug version 1.0-dev

Humbug running test suite to generate logs and code coverage data...

  361 [==========================================================] 28 secs

Humbug has completed the initial test run successfully.
Tests: 361 Line Coverage: 64.86%

Humbug is analysing source files...

Mutation Testing is commencing on 78 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)

.....M.M..EMMMMMSSSSMMMMMSMMMMMSSSE.ESSSSSSSSSSSSSSSSSM