Consider the off-line larning case of: \(min E=\frac{1}{P} \sum_{p=1}^P E_p \ \text{with}\ E_p = \frac{1}{2}\sum_{i=1}^{N_L}(x_{i,p}^{desired} - x_{i,p}^L)^2\) as the unconstrained nonlinear optimization \(min f(x)\) where \(f\) denotes the cost function and \(x\) the interconnection weights. The simplest optimization algorithm is steepest descent algorithm:
\[x_{k+1} = x_k - \alpha_k \nabla f(x_k)\]with \(x_k\) is the \(k\)-th iterate. In this case the search direction corresponds to minus the gradient of the cost function. However in the area of Local optimization algorithms more advanced methods exist which are much faster.
Faster methods: