FriconiX
Free collection of beautiful vector icons for your web pages.

Gradient descent example

Let's consider the function \(f: \mathbb{R^2} \mapsto \mathbb{R} \) given by:

$$ f(x,y) = (x-2)^2 + 2(y-3)^2 $$

Here is a 3D surface plot of this function:

Function to minize in our gradient descent algorithm

Optimization

We want to apply the gradient descent algorithm to find the minima. Steps are given by the following formula:

$$ X_{n+1} = X_n - \alpha \nabla f(X_n) $$

Let's start by calculating the gradient of \( f(x,y) \):

$$ \nabla f(X) = \begin{pmatrix} \frac{df}{dx} \\ \frac{df}{dy} \end{pmatrix} = \begin{pmatrix} 2x-4 \\ 4y-12 \end{pmatrix} $$

The coordinates will be updated according to:

$$ x_{n+1} = x_{n} - \alpha(2x_{n} - 4) $$ $$ y_{n+1} = y_{n} - \alpha(4y_{n} - 12) $$

In the following example, we arbitrary placed the starting point at coordinates \( X_0=(30,20) \). Here is an illustration of the convergence to \( X_200=(2,3) \) after 200 iterations:

Gradient descent path example

Download & source code

See also


Last update : 03/05/2020