The Numenta Anomaly Benchmark (NAB)

Welcome. This repository contains the data and scripts which comprise the Numenta Anomaly Benchmark (NAB) v1.1. NAB is a novel benchmark for evaluating algorithms for anomaly detection in streaming, real-time applications. It is composed of over 50 labeled real-world and artificial timeseries data files plus a novel scoring mechanism designed for real-time applications.

Included are the tools to allow you to run NAB on your own anomaly detection algorithms; see the NAB entry points info. Competitive results tied to open source code will be posted on the Scoreboard. Let us know about your work by emailing us at nab@numenta.org or submitting a pull request.

This readme is a brief overview and contains details for setting up NAB. Please refer to the following for more details about NAB scoring, data, and motivation:

Unsupervised real-time anomaly detection for streaming data - The main paper, covering NAB and Numenta's HTM-based anomaly detection algorithm
NAB Whitepaper
Evaluating Real-time Anomaly Detection Algorithms - Original publication of NAB

We encourage you to publish your results on running NAB, and share them with us at nab@numenta.org. Please cite the following publication when referring to NAB:

Ahmad, S., Lavin, A., Purdy, S., & Agha, Z. (2017). Unsupervised real-time anomaly detection for streaming data. Neurocomputing, Available online 2 June 2017, ISSN 0925-2312, https://doi.org/10.1016/j.neucom.2017.04.070

Scoreboard

The NAB scores are normalized such that the maximum possible is 100.0 (i.e. the perfect detector), and a baseline of 0.0 is determined by the "null" detector (which makes no detections).

| Detector | Standard Profile | Reward Low FP | Reward Low FN | |---------------|------------------|---------------|---------------| | Perfect | 100.0 | 100.0 | 100.0 | | ARTime | 74.9 | 65.1 | 80.4 | | Numenta HTM* | 70.5-69.7 | 62.6-61.7 | 75.2-74.2 | | CAD OSE† | 69.9 | 67.0 | 73.2 | | earthgecko Skyline | 58.2 | 46.2 | 63.9 | | KNN CAD† | 58.0 | 43.4 | 64.8 | | Relative Entropy | 54.6 | 47.6 | 58.8 | | Random Cut Forest **** | 51.7 | 38.4 | 59.7 | | Twitter ADVec v1.0.0| 47.1 | 33.6 | 53.5 | | Windowed Gaussian | 39.6 | 20.9 | 47.4 | | Etsy Skyline | 35.7 | 27.1 | 44.5 | | Bayesian Changepoint** | 17.7 | 3.2 | 32.2 | | EXPoSE | 16.4 | 3.2 | 26.9 | | Random*** | 11.0 | 1.2 | 19.5 | | Null | 0.0 | 0.0 | 0.0 |

As of NAB v1.0

* From NuPIC version 1.0 (available on PyPI); the range in scores represents runs using different random seeds.

** The original algorithm was modified for anomaly detection. Implementation details are in the detector's code.

*** Scores reflect the mean across a range of random seeds. The spread of scores for each profile are 7.95 to 16.83 for Standard, -1.56 to 2.14 for Reward Low FP, and 11.34 to 23.68 for Reward Low FN.

**** We have included the results for RCF using an AWS proprietary implementation; even though the algorithm code is not open source, the algorithm description is public and the code we used to run NAB on RCF is open source.

† Algorithm was an entry to the 2016 NAB Competition.

Please see the wiki section on contributing algorithms for discussion on posting algorithms to the scoreboard.

Corpus

The NAB corpus of 58 timeseries data files is designed to provide data for research in streaming anomaly detection. It is comprised of both real-world and artifical timeseries data containing labeled anomalous periods of behavior.

The majority of the data is real-world from a variety of sources such as AWS server metrics, Twitter volume, advertisement clicking metrics, traffic data, and more. All data is included in the repository, with more details in the data readme. Please contact us at nab@numenta.org if you have similar data (ideally with known anomalies) that you would like to see incorporated into NAB.

The NAB version will be updated whenever new data (and corresponding labels) is added to the corpus or other significant changes are made.

Additional Scores

For comparison, here are the NAB V1.0 scores for some additional flavors of HTM.

Numenta HTM using NuPIC v.0.5.6: This version of NuPIC was used to generate the data for the paper mentioned above (Unsupervised real-time anomaly detection for streaming data. Neurocomputing, ISSN 0925-2312, https://doi.org/10.1016/j.neucom.2017.04.070). If you are interested in replicating the results shown in the paper, use this version.
HTM Java is a Community-Driven Java port of HTM.
nab-comportex is a twist on HTM anomaly detection using Comportex, a community-driven HTM implementation in Clojure. Please see Felix Andrew's blog post on experiments with this algorithm.
NumentaTM HTM detector uses the implementation of temporal memory found here.
Numenta HTM detector with no likelihood uses the raw anomaly scores directly. To run without likelihood, set the variable self.useLikelihood in numenta_detector.py to False.

| Detector |Standard Profile | Reward Low FP | Reward Low FN | |---------------|---------|------------------|---------------| | Numenta HTMusing NuPIC v0.5.6* | 70.1 | 63.1 | 74.3 | | nab-comportex† | 64.6 | 58.8 | 69.6 | | NumentaTM HTM* | 64.6 | 56.7 | 69.2 | | HTM Java | 56.8 | 50.7 | 61.4 | | Numenta HTM*, no likelihood | 53.62 | 34.15 | 61.89 |

* From NuPIC version 0.5.6 (available on PyPI).

† Algorithm was an entry to the 2016 NAB Competition.

Installing NAB

Supported Platforms

OSX 10.9 and higher
Amazon Linux (via AMI)

Other platforms may work. NAB has been tested on Windows 10 but is not officially supported.

Initial requirements

You need to manually install the following:

Download this repository

Use the Github links provided in the right sidebar.

Install NAB

Pip:

From inside the checkout directory:

pip install -r requirements.txt
  pip install . --user

If you want to manage dependency versions yourself, you can skip dependencies with:

pip install . --user --no-deps

If you are actively working on the code and are familiar with manual PYTHONPATH setup:

  pip install -e . --install-option="--prefix=/some/other/path/"

Anaconda:

conda env create

Usage

There are several different use cases for NAB:

If you want to look at all the results we reported in the paper, there is no need to run anything. All the data files are in the data subdirectory and all individual detections for reported algorithms are checked in to the results subdirectory. Please see the README files in those locations.
If you want to plot some of the results, please see the README in the scripts directory for scripts/plot.py
If you have your own algorithm and want to run the NAB benchmark, please see the NAB Entry Points section in the wiki. (The easiest option is often to simply run your algorithm on the data and output results in the CSV format we specify. Then run the NAB scor

NAB

Install / Use

README