MSAD

<p align="center"> <img width="150" src="./assets/figures/MSAD.png"/> </p> <h1 align="center">Choose Wisely </h1> <h2 align="center">An Extensive Evaluation of Model Selection for Anomaly Detection in Time Series</h2> <div align="center"> <p> <img alt="GitHub" src="https://img.shields.io/github/license/boniolp/MSAD"> <img alt="GitHub issues" src="https://img.shields.io/github/issues/boniolp/MSAD"> </p> </div> <p align="center"><a href="https://adecimots.streamlit.app/"><strong>Adecimo</strong>: Explore our results and test with your own data</a></p>

MSAD proposes a pipeline for model selection based on time series classification and an extensive experimental evaluation of existing classification algorithms for this new pipeline. Our results demonstrate that model selection methods outperform every single anomaly detection method while being in the same order of magnitude regarding execution time. You can click on our demo link above to get more information and navigate through our experimental evaluation.

If you use MSAD in your project or research, please cite the following paper:

• VLDB, 2023

Extensions and Further Work

Journal Extension and MSADv2

This repository has been extended in the following work, which introduces weighted averaging techniques and more advanced model selection methods for anomaly detection in time series:

• MSAD: A deep dive into model selection for time series anomaly detection
  The VLDB Journal, 2025

  @article{sylligardos2025msad,
    title={MSAD: A deep dive into model selection for time series anomaly detection},
    author={Sylligardos, Emmanouil and Paparrizos, John and Palpanas, Themis and Senellart, Pierre and Boniol, Paul},
    journal={The VLDB Journal},
    volume={34},
    number={6},
    pages={1--25},
    year={2025},
    publisher={Springer}
  }
  

See the new repository:
sylligardos/MSADv2

Reference

Sylligardos, Emmanouil, et al. "Choose Wisely: An Extensive Evaluation of Model Selection for Anomaly Detection in Time Series." Proceedings of the VLDB Endowment 16.11 (2023): 3418-3432.

@article{sylligardos2023choose,
  title={Choose Wisely: An Extensive Evaluation of Model Selection for Anomaly Detection in Time Series},
  author={Sylligardos, Emmanouil and Boniol, Paul and Paparrizos, John and Trahanias, Panos and Palpanas, Themis},
  journal={Proceedings of the VLDB Endowment},
  volume={16},
  number={11},
  pages={3418--3432},
  year={2023},
  publisher={VLDB Endowment}
}

Contributors

• Emmanouil Sylligardos (ENS, Inria, DIENS, PSL, CNRS)
• Paul Boniol (ENS, Inria, DIENS, PSL, CNRS)

Installation

To install MSAD from source, you will need the following tools:

• git
• conda (anaconda or miniconda)

Steps for installation

Step 1: Clone this repository using git and change into its root directory.

git clone https://github.com/boniolp/MSAD.git
cd MSAD/

Step 2: Download the required datasets and weights from the following links (due to upload size limitations on GitHub, we host them on Google Drive):

• Download datasets

  • Move the downloaded file TSB.zip into the MSAD/data/ directory and unzip it.

• Download datasets + preprocesed data

  • Unzip the downloaded data.zip file and move its contents into the MSAD/data/ directory. With this files you can skip the steps of generating the Oracles, and creating the proccesed datasets (windowed and features).

• Download weights

  • Unzip the downloaded weights.zip file and move its contents (supervised/ and unsupervised/) into the MSAD/results/weights/ directory.

Step 3: Create and activate a conda environment named MSAD.

conda env create --file environment.yml
conda activate MSAD

Note: If you plan to use GPU acceleration, please ensure that you have CUDA installed. You can refer to the CUDA installation instructions for guidance.

If you do not wish to create the conda environment, you can install only the dependencies listed in requirements.txt using the following command:

pip install -r requirements.txt

4. :clap: Installation complete! :clap:

Usage

Below, you will find a step-by-step guide on how to use our work. This includes the commands required to run the scripts along with a small explanation of what they do and the parameters they use. The values of the parameters in the scripts are just examples, and you can experiment with different values.

Compute Oracle

The Oracle (shown in white in the results figure at the end) is a hypothetical model that simulates the accuracy of a model on a given benchmark and evaluates its anomaly detection ability. You can simulate Oracle with different accuracy values, ranging from 1 (always selecting the best detector for a time series) to zero (always selecting a wrong detector). Additionally, you can simulate Oracle with different modes of randomness, namely:

1. true: When wrong, randomly select another detector.
2. lucky: When wrong, always select the second best detector (upper bound).
3. unlucky: When wrong, always select the worst detector (lower bound).
4. best-k: When wrong, always select the k-th best detector (e.g., best-2 is lucky).

To compute Oracle, run the following command:

python3 run_oracle.py --path=data/TSB/metrics/ --acc=1 --randomness=true

• path: Path to metrics (the results will be saved here).
• acc: The accuracy that you want to simulate (a float between 0 and 1).
• randomness: The randomness mode that you want to simulate (see possible modes above).

The results are saved in /MSAD/data/TSB/metrics/TRUE_ORACLE-100/ (the name of the last folder should change depending on the parameters).

Compute Averaging Ensemble

The Averaging Ensemble, or Avg Ens (in orange in the results figure at the end), is used to ensemble the anomaly scores produced by all the detectors, by computing their average. Subsequently, the AUC-PR and the VUS-PR metrics are computed for the resulting score.

To compute Avg Ens, run the following command:

python3 run_avg_ens.py --n_jobs=16

• n_jobs: The number of threads to use for parallel computation (specify an appropriate value).

This process may take some time :smile: (~16 mins in 16 cores and 32GB of RAM). The script will perform the following tasks:

1. Load all datasets from the TSB benchmark
2. Load all the scores for each time series and detector (~ 1800 * 12 scores)
3. Compute the average score for each time series for 4 metrics (AUC-ROC, AUC-PR, VUS-ROC, VUS-PR)
4. Save the results in /MSAD/data/TSB/metrics/AVG_ENS/.

Data Preprocessing

Our models have been implemented to work with fixed-size inputs. Thus, before running any models, we first divide every time series in the TSB benchmark into windows. Note that you can add your own time series here and divide them into windows, but make sure to follow the same format.

To produce a windowed dataset, run the following command:

python3 create_windows_dataset.py --save_dir=data/ --path=data/TSB/data/ --metric_path=data/TSB/metrics/ --window_size=512 --metric=AUC_PR

• save_dir: Path to save the dataset.
• path: Path of the dataset to divide into windows.
• metric_path: Path to the metrics of the dataset provided (to produce the labels).
• window_size: Window size (if the window size is larger than the time series' length, that time series is skipped).
• metric: Metric to use for producing the labels (AUC-PR, VUS-PR, AUC-ROC, VUS-ROC).

The feature-based methods require a set of features to be computed first, turning the time series into tabular data. To achieve this, we use the TSFresh module, which computes a predefined set of features.

To compute the set of features for a segmented dataset, run the following command:

python3 generate_features.py --path=data/TSB_512/

• path: Path to the dataset for computing the features (the dataset should be segmented first into windows; see the command above). The resulting dataset is saved in the same directory (MANDATORY).

Note: This process is memory-intensive, and we required a minimum of 512GB of RAM to run it. If you encounter memory issues, optimizing the TSFresh library is not within the scope of this project.

Deep Learning Architectures

To train a model, run the following command:

python3 train_deep_model.py --path=data/TSB_512/ --split=0.7 --file=experiments/supervised_splits/split_TSB_512.csv --model=resnet --params=models/configuration/resnet_default.json --batch=256 --epochs=10 --eval-true

• path: Path to the dataset to use.
• split: Split percentage for training and validation sets.
• seed: Seed for train/val split (optional).
• file: Path to a file that contains a specific split (to reproduce our results).
• model: Model to use (type of architecture).
• params: A JSON file with the model's parameters.
• batch: Batch size.
• epochs: Number of training epochs.
• eval-true: Whether to evaluate the model on test data after training.

This script will save the following:

• training specific information into /MSAD/results/done_training/resnet_default_512_11092023_162841.csv file.
• TensorBoard data will be saved into /MSAD/results/runs/resnet_default_512_11092023_162726/.
• The trained weights will be saved in /MSAD/results/weights/resnet_default_512/.
• In case the 'eval-true' is selected, the results of the trained model on the test set will be saved in _/MSAD/results