Survey on Normalization Techniques

This repo is for our paper survey paper on normalization techniques in training deep neural networks:

Normalization Techniques in Training DNNs: Methodology, Analysis and Application,

Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu and Ling Shao.

arXiv preprint arXiv:2009.12836

We hope this repo provide a more friendly way for readers to review/follow the related works.

=========================Update: ===============================

2021-06-18. V2: Update for the Tutorial of Normalization

2020-11-28. V1: The initial version for the Survey paper

Table of content

• 1. Methodology
  • 1.1 Normalizing Activations by Population Statistics
  • 1.2 Normalizing Activations as Functions
    • 1.2.1 Normalization Area Partitioning
    • 1.2.2 Normalization Operation
    • 1.2.3 Normalization Representation Recovery
    • 1.2.4 Multi-Mode
    • 1.2.5 Combinational Normalization
    • 1.2.6 BN for More Robust Estimation
  • 1.3 Normalizing Weights
  • 1.4 Normalizing Gradients
• 2 Analysis
  • 2.1 Scale Invariance in Stabilizing Training
  • 2.2 Improved Conditioning in Optimization
  • 2.3 Stochasticity for Generalization
  • 2.4 Representation
• 3. Application
  • 3.1 Domain Adaptation
  • 3.2 Style Transfer
  • 3.3 Training GANs
  • 3.4 Efficient Deep Models
  • 3.5 Meta learning
  • 3.6 Reinforcement learning
  • 3.7 Unsupervised/Semi-supervised representation learning
  • 3.8 Graph Neural Networks
  • 3.8 Applied in miscellaneous networks
• Contact

---

Methodology

Normalizing Activations by Population Statistics

• Efficient BackProp. Neural Networks: Tricks of the Trade, 1998. paper.

• Accelerated Gradient Descent by Factor-Centering Decomposition. Technical Report, 1998. paper .

• Deep Boltzmann Machines and the Centering Trick. Neural Networks: Tricks of the trade, 2012. paper.

• Deep learning made easier by linear transformations in perceptrons. AISTATS, 2012. paper .

• Mean-normalized stochastic gradient for large-scale deep learning. ICASSP, 2014. paper.

• Natural Neural Networks. NeurIPS, 2015. paper.

• Learning Deep Architectures via Generalized Whitened Neural Networks. ICML, 2017. paper

Normalizing Activations as Functions

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML, 2015. paper , code.
• Knowledge matters: Importance of prior information for optimization. JMLR, 2016. paper .
• Recurrent Batch Normalization. ICLR, 2017. paper , code.
• Batch normalized recurrent neural networks. ICASSP, 2016. paper.

Normalization Area Partitioning

• Layer Normalization. arXiv:1607.06450, 2016. paper .
• Group Normalization. ECCV, 2018. paper , code.
• Instance Normalization: The Missing Ingredient for Fast Stylization. arXiv:1607.08022, 2016. paper , code.
• Positional Normalization. NeurIPS, 2019. paper , code.
• Four Things Everyone Should Know to Improve Batch Normalization. ICLR, 2020. paper , code.
• Local Context Normalization: Revisiting Local Normalization. CVPR, 2020. paper , code.
• What is the best multi-stage architecture for object recognition?. ICCV, 2009. paper .
• ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS, 2012. paper .
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes. ICLR, 2017. paper , code.

Normalization Operation

• Decorrelated Batch Normalization. CVPR, 2018. paper , code.
• Iterative Normalization: Beyond Standardization towards Efficient Whitening. CVPR, 2019. paper , code.
• Whitening and Coloring transform for GANs. ICLR, 2019. paper , code.
• An Investigation into the Stochasticity of Batch Whitening. CVPR, 2020. paper , code.
• Network Deconvolution. ICLR, 2020. paper , code.
• Channel Equilibrium Networks for Learning Deep Representation. ICML, 2020. paper , code.
• Concept Whitening for Interpretable Image Recognition. arXiv:2002.01650, 2020. paper , code.
• IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization. arXiv:2005.02178, 2020. paper.
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning. arXiv:1610.06160, 2016. paper.
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks. arXiv:1802.09769, 2018. paper .
• Norm matters: efficient and accurate normalization schemes in deep networks. NeurIPS, 2018. paper , code.
• Generalized Batch Normalization: Towards Accelerating Deep Neural Networks. AAAI, 2019. paper .
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. NeurIPS, 2016. paper , code.
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization. ICLR, 2020. paper , code.
• PowerNorm: Rethinking Batch Normalization in Transformers. ICML, 2020. paper , code.
• Progressive Growing of GANs for Improved Quality, Stability, and Variation. ICLR, 2018. paper , code.
• Root Mean Square Layer Normalization. NeurIPS, 2019. paper , code.
• Online Normalization for Training Neural Networks. NeurIPS, 2019. paper , code.
• Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction. arXiv:2006.12753, 2020. paper.

Normalization Representation Recovery

• Whitening and Coloring transform for GANs. ICLR, 2019. paper , code.

• Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition. INTERSPEECH, 2017. paper.

• Multimodal Unsupervised Image-to-Image Translation. ECCV, 2018. paper , code.

• U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation. ICLR, 2020. paper , code.

• Instance-Level Meta Normalization. CVPR, 2019. paper , code.

• Semantic Image Synthesis with Spatially-Adaptive Normalization. CVPR, 2019. paper , code.

• Instance Enhancement Batch Nor