Flat Spot Problem
The Flat Spot Problem is an issue observed by Scott Fahlman in a paper on the Quickprop
training method. Flat spot occurs in certain activation functions, particularly the
Sigmoid/Logistic Activation Function. Encog does not use flat spot on the Hyperbolic
Tangent or ReLU Activation Functions. Flatspot can make it very difficult for a neural
network to property train with propagation training. Because of the Flatspot problem,
certain hidden neurons can be rendered completely useless. This can greatly increase
training time, and decrease overall efficiency for neural networks. For small neural
networks, with just a few hidden neurons, if enough random weights fall into the flat
spot range, the neural network will fail to ever converge.
To see why the Flatspot problem exists, consider that all propagation training methods
require a derivative of the activation function. Consider the derivative of the sigmoid
activation function.
$$ o_j(1 − o_j) $$
Where $o_j$ is the sigmoid output from unit $j$. The above derivative will approach zero
when $o_j$ is near 1.0 or 0.0. The graph illustrates this. You can see the flat spot at
the top of the graph, near zero.
Eliminating the flat spot is as simple as adding a constant, such as 0.1 to the derivative
function. This results in.
$$ oj(1 − o_j) + 0.1 $$
This, generally, has a very positive effect on all propagation training. No change is
made to the actual activation function, so the flat spot modification is only necessary
at training time.
Encog Handling of the Flat Spot
By default Encog addresses the flat spot. This has been shown to enhance Encog training.
However, you can disable the flat spot propcessing. To do this set the FixFlatSpot
property, on any propagation training object to false.
References
- An Empirical Study of Learning Speed in Back-Propagation Networks” (Scott E. Fahlman, 1988)