Let's consider the function \(f: \mathbb{R^2} \mapsto \mathbb{R} \) given by:

$$ f(x,y) = (x-2)^2 + 2(y-3)^2 $$

Here is a 3D surface plot of this function:

We want to apply the gradient descent algorithm to find the minima. Steps are given by the following formula:

$$ X_{n+1} = X_n - \alpha \nabla f(X_n) $$

Let's start by calculating the gradient of \( f(x,y) \):

$$ \nabla f(X) = \begin{pmatrix} \frac{df}{dx} \\ \frac{df}{dy} \end{pmatrix} = \begin{pmatrix} 2x-4 \\ 4y-12 \end{pmatrix} $$

The coordinates will be updated according to:

$$ x_{n+1} = x_{n} - \alpha(2x_{n} - 4) $$ $$ y_{n+1} = y_{n} - \alpha(4y_{n} - 12) $$

In the following example, we arbitrary placed the starting point at coordinates \( X_0=(30,20) \). Here is an illustration of the convergence to \( X_{200}=(2,3) \) after 200 iterations:

- Gradient descent example in Matlab
- Gradient descent example in Python
- Gradient descent example in C/C++

- Neural networks curve fitting
- Datasets for deep learning
- How popular are neural networks over the years?
- Install TensorFlow and Keras for Linux
- Learning rule demonstration
- Linear regression example
- Most popular activation functions for deep learning
- Most relevant deep learning research papers
- Neural Network Perceptron
- Simplest neural network with TensorFlow
- Simplest perceptron
- Single layer training algorithm
- Single layer classification example
- Gradient descent for neural networks
- Single layer limitations
- Neural networks

Last update : 01/30/2021