At the end of the last lecture, we obtained the following lemma on the gradient descent for smooth functions.

However, it is still not easy to show the convergence rate for the gradient descent. We now introduce a continuous version of the gradient descent instead, which is easier to analyse.

Applying the chain rule, is decreasing since Now we can take the derivative by convexity. Then integrating both sides, we obtain that which further gives that

We compare the gradient descent with the gradient flow. Assume the gradient descent iterates with a fixed step size and an initial point , i.e., For the gradient descent, For the gradient flow, Intuitively we know that, if does not change too fast, the gradient descent approximates the gradient flow.

If we hope , we need to run the gradient descent steps. If the initial point is far from , and is sufficiently small, the gradient descent is slow. Unfortunately, consider the following function
This function is convex and -smooth. Hence and the convergence rate of the gradient descent will be very small if .

Recall that, if we run the gradient descent for a quadratic function where , it gives that and thus . Clearly converges to the optimal value at an exponential rate.

We now introduce the following definition, which requires the function is a bit "better" than some quadratic function.

There are some other forms of quadratic functions. Why don’t we choose other functions such as or for some given ? In fact, these functions mentioned can just achieve a similar effect to . For example, is almost equivalent to . In addition, Hence, all quadratic functions achieve similar effects to .

Recall that, a function is convex iff its hessian matrix is positive semidefinite. The hessian matrix of is

We also have the following lemma similar to the first order condition for convexity and smoothness.

As a corollary, above lemma implies that for any . Hence, is strictly convex.

Recall the property of monotone gradient for convex functions. We have a similar corollary.

We now establish the convergence of gradient descent with strong convexity. First consider the gradient flow again. By strong convexity, we can bound the derivative as follows For a time-continuous non-negative process , if , then we have . The same result holds if we replace the equality by an inequality.

Applying the Gronwall’s lemma, we conclude immediately, which gives an exponential decay rate. Intuitively, as the discretization version of the gradient flow, the gradient descent for strongly convex functions should also follow the exponential decay.

The function value also has an exponential decay. Since is -smooth, we have which gives the following corollary.

For a quadratic function where , we already have the following facts:

• is -strongly convex;
• is -smooth.

Applying the above theorem, if we take the step size , will converge at an exponential rate of (since ). Recall the in last lecture, we have shown as long as , which means that is not necessary for convergence. Since we hope is as small as possible, we may choose a greater value of to obtain a better rate.

Now let us calculate the optimal convergence rate of . Since , we have
Applying the eigen-decomposition , where and .
Then where . So Let . We have .

Note that . Thus, We would like to choose to minimize , which means . In this case,

The argument above reveals that for quadratic functions, the convergence rate of gradient descent depends on .

For nonquadratic functions, we can approximate them locally (near the minimum point ) by the following Taylor series Hence, in the neighborhood of the minimum point , the convergence rate depends on . If the condition number of is large, the results given by gradient descent with fixed step size cannot converge rapidly.

Here are some well-conditioned and ill-conditioned examples: