Oodeel
Simple, compact, and hackable post-hoc deep OOD detection for already trained tensorflow or pytorch image classifiers.
Install / Use
/learn @deel-ai/OodeelREADME
<b>Oodeel</b> is a library that performs post-hoc deep OOD (Out-of-Distribution) detection on already trained neural network image classifiers. The philosophy of the library is to favor quality over quantity and to foster easy adoption. As a result, we provide a simple, compact and easily customizable API and carefully integrate and test each proposed baseline into a coherent framework that is designed to enable their use in tensorflow and pytorch. You can find the documentation here.
from oodeel.methods import MLS
mls = MLS()
mls.fit(model) # A tensorflow or torch model
scores, info = mls.score(ds) # ds is a tf.data.Dataset or a torch.DataLoader
Table of contents
- Installation
- Quick Start
- Tutorials
- What's Included
- Development roadmap
- Contributing
- See Also
- Acknowledgments
- Creators
- Citation
- License
Installation
Installation can be done using:
pip install oodeel
oodeel requires either tensorflow or pytorch to be already installed (it will not install them automatically not to mess-up with existing installations). It is regularly tested with:
|Python version|Pytorch version|Tensorflow version|
|---|---|---|
|3.9| 2.0| 2.11|
|3.10|2.4 | 2.13|
|3.11| 2.6 | 2.15|
Quick Start
Now that oodeel is installed, here are some basic examples of what you can do with the available modules. See also the notebooks directory for more advanced examples.
For benchmarking with one dataset as in-distribution and another as out-of-distribution
Load in-distribution and out-of-distribution datasets.
from oodeel.datasets import load_data_handler
data_handler = load_data_handler("torch")
# use backend="tensorflow" if using tensorflow
# Do not forget to adapt load_kwargs in that case
# Load in-distribution dataset: CIFAR-10
ds_in = data_handler.load_dataset(
"CIFAR10", load_kwargs={"root": data_path, "train": False, "download": True}
)
# Load out-of-distribution dataset: SVHN
ds_out = data_handler.load_dataset(
"SVHN", load_kwargs={"root": data_path, "split": "test", "download": True}
)
# Prepare datasets for forward (requires appropriate preprocess_fn e.g. input normalization)
ds_in = data_handler.prepare(
ds_in, batch_size, preprocess_fn, columns=["input", "label"]
)
ds_out = data_handler.prepare(
ds_out, batch_size, preprocess_fn, columns=["input", "label"]
)
For benchmarking with a classes subset as in-distribution and another classes subset as out-of-distribution
Load a dataset and split it into an in-distribution dataset and ou-of-distribution dataset depending on its label values (a common practice of anomaly detection and open set recognition).
from oodeel.datasets import load_data_handler
data_handler = load_data_handler("torch") # use backend="tensorflow" if using tensorflow
# Load dataset to split: CIFAR-10
ds_test = data_handler.load_dataset(
"CIFAR10", load_kwargs={"root": data_path, "train": False, "download": True}
)
in_labels = [0, 1, 2, 3, 4]
ds_in, ds_out = data_handler.split_by_class(ds_test, in_labels)
# Prepare datasets for forward (requires appropriate preprocess_fn e.g. input normalization)
ds_in = data_handler.prepare(
ds_in, batch_size, preprocess_fn, columns=["input", "label"]
)
ds_out = data_handler.prepare(
ds_out, batch_size, preprocess_fn, columns=["input", "label"]
)
Run an OOD method
Load an OOD method and use it on an already-trained model
from oodeel.methods import MLS
mls = MLS()
mls.fit(model) # Requires a pretrained model
# info_X contains model predictions and labels if available
scores_in, info_in = mls.score(ds_in)
scores_out, info_out = mls.score(ds_out)
Evaluate the method
from oodeel.eval.metrics import bench_metrics
metrics = bench_metrics(
(scores_in, scores_out),
metrics = ["auroc", "fpr95tpr"],
)
And visualize the results!
2D t-SNE (3D is also available).
plot_2D_features(
model=model,
in_dataset=ds_in,
out_dataset=ds_out,
output_layer_id=-2,
)
Classical histograms and AUROC curve.
plot_ood_scores(scores_in, scores_out, log_scale=False)
plot_roc_curve(scores_in, scores_out)
Tutorials
We propose some tutorials to get familiar with the library and its API. See the Tutorial section of the doc
What's Included
The library is based on a class, OODBaseDetector, that fits a model and then scores new samples. Some baselines use extra data, so OODBaseDetector can also fit additional data if needed. The library uses OODDataset to properly load data from different sources and prepare it for OOD detection. It can perform OOD-specific operations like adding extra OOD data for tuning with Outlier Exposure or filters according to label values for anomaly detection or open set recognition benchmarks.
Currently, oodeel includes the following baselines:
| Name | Link | Venue | Status | | --- | --- | --- | --- | | MLS | Open-Set Recognition: a Good Closed-Set Classifier is All You Need? | ICLR 2022 | avail tensorflow & torch | | MSP | A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks | ICLR 2017 | avail tensorflow & torch| | Mahalanobis | A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks | NeurIPS 2018 | avail tensorflow or torch| | Energy | Energy-based Out-of-distribution Detection | NeurIPS 2020 |avail tensorflow or torch | | Odin | Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks | ICLR 2018 | avail tensorflow or torch | | DKNN | Out-of-Distribution Detection with Deep Nearest Neighbors | ICML 2022 | avail tensorflow or torch | | VIM | ViM: Out-Of-Distribution with Virtual-logit Matching | CVPR 2022 |avail tensorflow or torch | | Entropy | Likelihood Ratios for Out-of-Distribution Detection | NeurIPS 2019 |avail tensorflow or torch | | ReAct | ReAct: Out-of-distribution Detection With Rectified Activations | NeurIPS 2021 | avail tensorflow or torch | | Gram | Detecting Out-of-Distribution Examples with Gram Matrices | ICML 2020 | avail tensorflow or torch | | GEN | GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection | CVPR 2023 | avail tensorflow or torch | | RMDS | A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection | preprint | avail tensorflow or [torch](docs/noteboo
Related Skills
node-connect
349.2kDiagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps
frontend-design
109.5kCreate distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.
openai-whisper-api
349.2kTranscribe audio via OpenAI Audio Transcriptions API (Whisper).
qqbot-media
349.2kQQBot 富媒体收发能力。使用 <qqmedia> 标签，系统根据文件扩展名自动识别类型（图片/语音/视频/文件）。
