MNIST-baselines

This repository contains various baseline models for comparison on the polluted MNIST dataset

Requirements

• NumPy
• scikit-learn
• PyTorch

Usage

Create a model

1. Prepare Model_Name.json in config/
2. Prepare Model_Name.py in models/
3. Prepare Trainer_Name.py in trainers/ (optional)

Train a model

python3 main.py --method Model_Name

Test a model

python3 main.py --method Model_Name --test

Trained models

Available at https://drive.google.com/open?id=1jXe2kJJLkhGH8T30juGbyUgsaDQZXISu

Polluted MNIST

<figure class="half"> <img src="./figures/dataset_demo.png" width="100%"> </figure>

Feature Extraction

A visualization of the first 15 principle components:

<figure class="half"> <img src="./figures/pca_res.png" width="70%"> </figure>

| Model | Accuracy / % | Model | Accuracy / % | | :---- |:------------:| :---- |:------------:| | Plain DNN | 90.72 | DNN + Dropout | 90.74 | | DNN + Batch Normalization | 92.39 | DNN + PCA (0.99 variance) | 21.40 | | DNN + PCA (0.95 variance) | 21.34 | DNN + PCA (0.90 variance) | 20.55 | | DNN + ICA | 11.93 | DNN + NMF | 11.30 | | DNN + VAE | 95.24 |

Conventional Classification Approaches

| Model | Accuracy / % | Model | Accuracy / % | | :---- |:------------:| :---- |:------------:| | Empty Model | 11.04 | Naive Bayes | 19.14 | | Logistic Regression | 24.37 | Decision Tree (gini) | 51.24 | | Decision Tree (entropy) | 50.91 | KNN (3 neighbours) | 78.82 | | KNN (5 neighbours) | 78.46 | KNN (10 neighbours) | 76.40 | | SGD | 22.30 | SVM (sigmoid kernel) | 10.84 | | SVM (rbf kernel) | 85.94 | SVM (polynomial kernel) | 87.11 |

The training curves of the above DNN models:

<figure class="half"> <img src="./figures/dnn_accuracy.png" width="70%"> <img src="./figures/dnn_loss.png" width="70%"> </figure>

Deep Convolutional Neural Networks

| Model | Accuracy / % | Model | Accuracy / % | | :---- |:------------:| :---- |:------------:| | LeNet-5 | 98.35 | MobileNetV2 | 99.63 | | VGG-19 | 99.64 | ShuffleNetG3 | 99.65 | | ResNet-101 | 99.71 | GoogLeNet | 99.82 | | PreAct ResNet-152 | 99.68 | PNASNet | 99.75 | | DenseNet-161 | 99.75 | ResNeXt-29-8x64d | 99.71 | | DPN-92 | 99.76 | SENet-18 | 99.69 | | MobileNet | 99.44 | CapsNet | 98.84 |

<figure class="half"> <img src="./figures/cnn_accuracy.png" width="70%"> <img src="./figures/cnn_loss.png" width="70%"> </figure>

Error Analysis

<figure class="half"> <img src="./figures/faults.png" width="70%"> </figure>

Team Members

• Xiangyu Chen
• Zelin Ye
• Yiheng Zhang