بسم الله الرحمن الرحیم
پیاده سازی رسمی سیمپل نت در کفی 2016

Lets Keep it simple, Using simple architectures to outperform deeper and more complex architectures (2016).

GitHub Logo

This repository contains the architectures, Models, logs, etc pertaining to the SimpleNet Paper (Lets keep it simple: Using simple architectures to outperform deeper architectures ) : https://arxiv.org/abs/1608.06037

SimpleNet-V1 outperforms deeper and heavier architectures such as AlexNet, VGGNet,ResNet,GoogleNet,etc in a series of benchmark datasets, such as CIFAR10/100, MNIST, SVHN. It also achievs a higher accuracy (currently 72.03/90.32) in imagenet, more than VGGNet, ResNet, MobileNet, AlexNet, NIN, Squeezenet, etc with only 5.7M parameters. It also achieves 74.23/91.748) with 9m version.
Slimer versions of the architecture work very decently against more complex architectures such as ResNet, WRN and MobileNet as well.

Citation

If you find SimpleNet useful in your research, please consider citing:

@article{hasanpour2016lets,
  title={Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures},
  author={Hasanpour, Seyyed Hossein and Rouhani, Mohammad and Fayyaz, Mohsen and Sabokrou, Mohammad},
  journal={arXiv preprint arXiv:1608.06037},
  year={2016}
}

(Check the successor of this architecture at Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet)

Other Implementations :

<img src="https://github.com/pytorch/pytorch/blob/main/docs/source/_static/img/pytorch-logo-dark.png" width="86" height="17" /> Official Pytorch implementation

Results Overview :

ImageNet result below was achieved using the Pytorch implementation

| Dataset | Accuracy | |------------|----------| | ImageNet-top1 (9m) | 74.23 | | ImageNet-top1 (5m) | 72.03 | | Cifar10 | 95.51 | | CIFAR100* | 78.37| | MNIST | 99.75 | | SVHN | 98.21 |

Achieved using Pytorch implementation

ImageNet Result:

SimpleNet outperforms much deeper and larger architectures on the ImageNet dataset:

| Model | Params | Top1 | Top5 | | :--------------- | :--------: | :-------: | :------: | | AlexNet | 60M | 57.2 | 80.3 | | SqeezeNet | 1.2M | 58.18 | 80.62 | | VGGNet16 | 138M | 71.59 | 90.38 | | VGGNet16_BN | 138M | 73.36 | 91.52 | | VGGNet19 | 143M | 72.38 | 90.88 | | VGGNet19_BN | 143M | 74.22 | 91.84 | | GoogleNet | 6.6M | 69.78 | 89.53 | | WResNet18 | 11.7M | 69.60 | 89.07 | | ResNet18 | 11.7M | 69.76 | 89.08 | | ResNet34 | 21.8M | 73.31 | 91.42 | | SimpleNet_small_050 | 1.5M | 61.67 | 83.49 | | SimpleNet_small_075 | 3.2M | 68.51 | 88.15 | | SimpleNet_5m | 5.7M | 72.03 | 90.32 | | SimpleNet_9m | 9.5M | 74.23 | 91.75 |

Extended ImageNet Result:

| Model | #Params | ImageNet | ImageNet-Real-Labels | | :--------------------------- | :----------: | :-----------: | :------------------: |
| simplenetv1_9m_m2(36.3 MB) | 9.5m | 74.23 / 91.748 | 81.22 / 94.756 |
| simplenetv1_5m_m2(22 MB) | 5.7m | 72.03 / 90.324 | 79.328/ 93.714 |
| simplenetv1_small_m2_075(12.6 MB)| 3m | 68.506/ 88.15 | 76.283/ 92.02 |
| simplenetv1_small_m2_05(5.78 MB) | 1.5m | 61.67 / 83.488 | 69.31 / 88.195 |

SimpleNet performs very decently, it outperforms VGGNet, variants of ResNet and MobileNets(1-3)
and is pretty fast as well! and its all using plain old CNN!.
To view the full benchmark results visit the benchmark page.
To view more results checkout the the Pytorch implementation page

Top CIFAR10/100 results:

| Method | #Params | CIFAR10 | CIFAR100 | | :--------------------------- | :----------: | :-----------: | :----------: | | VGGNet(16L) /Enhanced | 138m | 91.4 / 92.45 | - | | ResNet-110L / 1202L * | 1.7/10.2m | 93.57 / 92.07 | 74.84/72.18 | | SD-110L / 1202L | 1.7/10.2m | 94.77 / 95.09 | 75.42 / - | | WRN-(16/8)/(28/10) | 11/36m | 95.19 / 95.83 | 77.11/79.5 | | Highway Network | N/A | 92.40 | 67.76 | | FitNet | 1M | 91.61 | 64.96 | | FMP* (1 tests) | 12M | 95.50 | 73.61 | | Max-out(k=2) | 6M | 90.62 | 65.46 | | Network in Network | 1M | 91.19 | 64.32 | | DSN | 1M | 92.03 | 65.43 | | Max-out NIN | - | 93.25 | 71.14 | | LSUV | N/A | 94.16 | N/A | | SimpleNet-Arch 1(۞) | 5.48M | 94.75 | - | | SimpleNet-Arch 2 (۩) | 5.48M | 95.51 | 78.37 |

*Note that the Fractional max pooling[13] uses deeper architectures and also uses extreme data augmentation.۞ means No zero-padding or normalization with dropout and ۩ means Standard data-augmentation- with dropout. To our knowledge, our architecture has the state of the art result, without aforementioned data-augmentations.

MNIST results:

| Method | Error rate | | :------------------------------------------- | :------------: | | DropConnect** | 0.21% | | Multi-column DNN for Image Classiﬁcation** | 0.23% | | APAC** | 0.23% | | Generalizing Pooling Functions in CNN** | 0.29% | | Fractional Max-Pooling** | 0.32% | | Batch-normalized Max-out NIN | 0.24% | | Max-out network (k=2) | 0.45% | | Network In Network | 0.45% | | Deeply Supervised Network | 0.39% | | RCNN-96 | 0.31% | | SimpleNet * | 0.25% |

*Note that we didn’t intend on achieving the state of the art performance here as we are using a single optimization policy without fine-tuning hyper parameters or data-augmentation for a specific task, and still we nearly achieved state-of-the-art on MNIST. **Results achieved using an ensemble or extreme data-augmentation

Top SVHN results:

| Method | Error rate | | :--------------------------- | :------------: | | Network in Network | 2.35 | | Deeply Supervised Net | 1.92 | | ResNet (reported by (2016)) | 2.01 | | ResNet with Stochastic Depth | 1.75 | | Wide ResNet | 1.64 | | SimpleNet | 1.79 |

Table 6-Slimmed version Results on Different Datasets

| Model | Ours | Maxout | DSN | ALLCNN | dasNet | ResNet(32) | WRN | NIN | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | #Param | 310K | 460K | 6M | 1M | 1.3M | 6M | 475K | 600K | 1M | | CIFAR10 | 91.98 | 92.33 | 90.62 | 92.03 | 92.75 | 90.78 | 91.6 | 93.15 | 91.19 | | CIFAR100 | 64.68|66.82|65.46|65.43|66.29|66.22|67.37|69.11|- |

| Other datasets | Our result | | --- | --- | | MNIST(310K)* | 99.72 | | SVHN(310K)* | 97.63 |

*Since we presented their results in their respective sections, we avoided mentioning the results here again.

Cifar10 extended results:

| Method | Accuracy | #Params | | :-------------------------------------- | :----------: | :----------: | | VGGNet(16L) | 91.4 | 138m | | VGGNET(Enhanced-16L)* | 92.45 | 138m | | ResNet-110* | 93.57 | 1.7m | | ResNet-1202 | 92.07 | 10.2m | | Stochastic depth-110L | 94.77 | 1.7m | | Stochastic depth-1202L | 95.09 | 10.2m | | Wide Residual Net | 95.19 | 11m | | Wide Residual Net | 95.83 | 36m | | Highway Network | 92.40 | - | | FitNet | 91.61 | 1M | | SqueezNet-(tested by us) | 79.58 | 1.3M | | ALLCNN | 92.75 | - | | Fractional Max-pooling* (1 tests) | 95.50 | 12M | | Max-out(k=2) | 90.62 | 6M | | Network in Network | 91.19 | 1M | | Deeply Supervised Network |

SimpleNet

Install / Use

README