Batch Normalization
Batch Normalization is a milestone technique in the development of deep learning, enabling various networks to train. However, BN’s error increases rapidly when the batch size becomes small affecting the batch statistics estimation.
Furthermore, the concept of “batch” is not always present.
For example, batch-wise normalization is not legitimate at inference time, so the mean and variance are pre-computed from the training set, often by running average. The pre-computed statistics may also change when the target data distribution changes. These issues lead to inconsistency in training, transferring, and testing time.
Alternatives
Here are several normalization methods proposed considering the issues above and their performance comparison using RESNET-50 evaluated with ImageNet.
