Mish

<h1 align="center">Mish: Self Regularized <br> Non-Monotonic Activation Function</h1> <p align="center"> <a href="LICENSE" alt="License"> <img src="https://img.shields.io/badge/License-MIT-brightgreen.svg" /></a> <a href="https://arxiv.org/abs/1908.08681v3" alt="ArXiv"> <img src="https://img.shields.io/badge/Paper-arXiv-blue.svg" /></a> <a href="https://scholar.googleusercontent.com/scholar.bib?q=info:j0C1gbodjP4J:scholar.google.com/&output=citation&scisdr=CgX0hbDMEOzUo74J6TM:AAGBfm0AAAAAX1QM8TNcu4tND6FEofKsXzM3cs1uCAAW&scisig=AAGBfm0AAAAAX1QM8Y5elaJ1IW-BKOuU1zFTYNp-QaNQ&scisf=4&ct=citation&cd=-1&hl=en" alt="Cite"> <img src="https://img.shields.io/badge/Cite-BibTex-blue.svg" /></a> <a href=" " alt="Citations"> <img src="https://img.shields.io/badge/Google Scholar-2855-lightgrey.svg" /></a> <a href="https://www.bmvc2020-conference.com/conference/papers/paper_0928.html" alt="Publication"> <img src="https://img.shields.io/badge/BMVC-2020-red.svg" /></a> <a href="https://console.paperspace.com/github/digantamisra98/Mish/blob/master/Layers_Acc.ipynb"> <img src="https://assets.paperspace.io/img/gradient-badge.svg" alt="Run on Gradient"/> </a> </p> <p align="center">BMVC 2020 <a href="https://www.bmvc2020-conference.com/assets/papers/0928.pdf" target="_blank">(Official Paper)</a></p> <br> <br> <details> <summary>Notes: (Click to expand)</summary>

• A considerably faster version based on CUDA can be found here - Mish CUDA (All credits to Thomas Brandon for the same)
• Memory Efficient Experimental version of Mish can be found here
• Faster variants for Mish and H-Mish by Yashas Samaga can be found here - ConvolutionBuildingBlocks
• Alternative (experimental improved) variant of H-Mish developed by Páll Haraldsson can be found here - H-Mish (Available in Julia)
• Variance based initialization method for Mish (experimental) by Federico Andres Lois can be found here - Mish_init

</details> <details> <summary>Changelogs/ Updates: (Click to expand)</summary>

• [07/17] Mish added to OpenVino - Open-1187, Merged-1125
• [07/17] Mish added to BetaML.jl
• [07/17] Loss Landscape exploration progress in collaboration with Javier Ideami and Ajay Uppili Arasanipalai <br>
• [07/17] Poster accepted for presentation at DLRLSS hosted by MILA, CIFAR, Vector Institute and AMII
• [07/20] Mish added to Google's AutoML - 502
• [07/27] Mish paper accepted to 31st British Machine Vision Conference (BMVC), 2020. ArXiv version to be updated soon.
• [08/13] New updated PyTorch benchmarks and pretrained models available on PyTorch Benchmarks.
• [08/14] New updated Arxiv version of the paper is out.
• [08/18] Mish added to Sony Nnabla - Merged-700
• [09/02] Mish added to TensorFlow Swift APIs - Merged - 1068
• [06/09] Official paper and presentation video for BMVC is released at this link.
• [23/09] CSP-p7 + Mish (multi-scale) is currently the SOTA in Object Detection on MS-COCO test-dev while CSP-p7 + Mish (single-scale) is currently the 3rd best model in Object detection on MS-COCO test dev. Further details on paperswithcode leaderboards.
• [11/11] Mish added to TFLearn - Merged 1159 (Follow up 1141)
• [17/11] Mish added to MONAI - Merged 1235
• [20/11] Mish added to plaidml - Merged 1566
• [10/12] Mish added to Simd and Synet - Docs
• [14/12] Mish added to OneFlow - Merged 3972
• [24/12] Mish added to GPT-Neo
• [21/04] Mish added to TensorFlow JS
• [02/05] Mish added to Axon
• [26/05] 🔥 Mish is added to PyTorch. Will be added in PyTorch 1.9. 🔥
• [27/05] Mish is added to PyTorch YOLO v3
• [09/06] 🔥 Mish is added to MXNet.
• [03/07] Mish is added to TorchSharp.
• [05/08] Mish is added to KotlinDL.

</details>

News/ Media Coverage:

• (02/2020): Podcast episode on Mish at Machine Learning Café is out now. Listen on:

<p float="center"> &emsp; &emsp; <a href="https://podcasts.apple.com/hu/podcast/mish-activation-function-with-diganta-misra-007/id1490681799?i=1000464407163" alt="Apple Podcasts"> <img src="podcast_logo/applepodcasts.png" width="150"/></a> <a href="https://open.spotify.com/episode/4sT9sxjSbAKtvJ6hTFg9zc" alt="Spotify"> <img src="https://github.com/digantamisra98/Mish/blob/master/podcast_logo/spotify.png" width="150"/></a> </p>

• (02/2020): Talk on Mish and Non-Linear Dynamics at Sicara is out now. Watch on:

<p float="center"> &emsp; &emsp; <a href="https://youtu.be/T2CRFROKcLM" alt="YouTube"> <img src="podcast_logo/yt1.png" width="100"/></a> </p>

• (07/2020): CROWN: A comparison of morphology for Mish, Swish and ReLU produced in collaboration with Javier Ideami. Watch on:

<p float="center"> &emsp; &emsp; <a href="https://www.youtube.com/watch?v=XRGu23hfzaQ" alt="YouTube"> <img src="podcast_logo/yt1.png" width="100"/></a> </p>

• (08/2020): Talk on Mish and Non-Linear Dynamics at Computer Vision Talks. Watch on:

<p float="center"> &emsp; &emsp; <a href="https://youtu.be/whOdg-yrgdI" alt="YouTube"> <img src="podcast_logo/yt1.png" width="100"/></a> </p>

• (12/2020): Talk on From Smooth Activations to Robustness to Catastrophic Forgetting at Weights & Biases Salon is out now. Watch on:

<p float="center"> &emsp; &emsp; <a href="https://www.youtube.com/watch?v=1U-7TWysqIg" alt="YouTube"> <img src="podcast_logo/yt1.png" width="100"/></a> <br> </p>

• (12/2020) Weights & Biases integration is now added 🔥. Get started.

• (08/2021) Comprehensive hardware based computation performance benchmark for Mish has been conducted by Benjamin Warner. Blogpost.

<br> <details> <summary><a href="https://dlrl.ca/"><b>MILA/ CIFAR 2020 DLRLSS</b></a> (Click on arrow to view)</summary> <div style="text-align:center"><img src ="poster_landscape-1.png" width="1000"/></div> </details> <br> <details> <summary><b>Contents</b>: (Click to expand)</summary>

1. Mish <br> a. Loss landscape
2. ImageNet Scores
3. MS-COCO
4. Variation of Parameter Comparison<br> a. MNIST<br> b. CIFAR10<br>
5. Significance Level <br>
6. Results<br> a. Summary of Results (Vision Tasks)<br> b. [Summary of Results (Language Tasks)](https://github.com/digantamisra98/Mish#summary-of-results-language-tas