Gradient descent example

Let's consider the function \(f: \mathbb{R^2} \mapsto \mathbb{R} \) given by:

$$ f(x,y) = (x-2)^2 + 2(y-3)^2 $$

Here is a 3D surface plot of this function:

Optimization

We want to apply the gradient descent algorithm to find the minima. Steps are given by the following formula:

$$ X_{n+1} = X_n - \alpha \nabla f(X_n) $$

Let's start by calculating the gradient of \( f(x,y) \):

$$ \nabla f(X) = \begin{pmatrix} \frac{df}{dx} \\ \frac{df}{dy} \end{pmatrix} = \begin{pmatrix} 2x-4 \\ 4y-12 \end{pmatrix} $$

The coordinates will be updated according to:

$$ x_{n+1} = x_{n} - \alpha(2x_{n} - 4) $$ $$ y_{n+1} = y_{n} - \alpha(4y_{n} - 12) $$

In the following example, we arbitrary placed the starting point at coordinates \( X_0=(30,20) \). Here is an illustration of the convergence to \( X_{200}=(2,3) \) after 200 iterations:

Python implementation

Download & source code

• Gradient descent example in Matlab
• Gradient descent example in Python
• Gradient descent example in C/C++

See also

• Neural networks curve fitting
• Datasets for deep learning
• How popular are neural networks over the years?
• Install TensorFlow and Keras for Linux
• Learning rule demonstration
• Linear regression example
• Most popular activation functions for deep learning
• Most relevant deep learning research papers
• Neural Network Perceptron
• Simplest neural network with TensorFlow
• Simplest perceptron
• Single layer training algorithm
• Single layer classification example
• Gradient descent for neural networks
• Single layer limitations
• Neural networks

---