Structure
Gaussian Error Unit is a high-performing neural network activation function that weights inputs by their value, rather than gates inputs by their sign as in ReLUs. GELU is defined as the equation in the image.
Results
GELU exceeds the accuracy of ELU and ReLU consistently in numerous datasets such as TIMIT FRAME and CIFAR-10.
