lec12

The gradient descent algorithm converges slowly if the condition number is large. For example, consider the function . At some point , . It locally decreases rapidly but not globally. In fact, the ideal descending direction is . So we would like to find a better descent direction.

Analogous to the root-finding Newton's method, where we use a Taylor series to estimate the objective function, we can approximate the objective function with a Taylor expansion, and (we hope that) the minimizer of this approximation then serves as an estimate for the true minimum. Given a convex function , consider its second-order Taylor approximation Denote by the right hand side. Now we compute the minimizer of . Since is also convex, it is equivalent to find a solution to . Note that if is invertible. Thus the minimizer of is . Then, we can update the iterate as the minimizer of the local quadratic approximation, namely, and repeat this process until convergence. This approach is called the Newton's method for optimization.

Note that if the objective function is strictly convex, and , then it follows that is invertible and . This is because if 's eigenvalues are all positive, their inverse are also positive. Therefore, .

Recall our requirement for the descending direction . We hope . Clearly, if , is a descending direction, since unless .

Intuitively this is not true since we use the second order Taylor series to approximate the function, but the Taylor series only works locally.

If the second-order Taylor series estimates the value of functions well, then given by the Newton's method is the minimum point of the Taylor series and it should be close to the minimum point of . But what happens if the Taylor series doesn't approximates well? Now we consider some "bad" examples.

The reason why the Newton's method does not converge in this example is that the second-order derivative of is close to zero when is large, which yields that the second-order Taylor series is not a good approximation near the minimum point. The Taylor series approximates well in the neighborhood of , but loses control at , if is large.

However, just keeping small is not enough yet. Here is another "bad" example.

In this example, the reason to failure is that changes rapidly so the second order Taylor series cannot approximates well even in the neighborhood of .

We now give some conditions to guarantee the convergence of the Newton’s method iterates.

Recall the gradient descent iteration , where we actually calculate the minimum point of the following function and let be its minimizer to approximate the minimum point of . In particular, if , is an upper estimation of by -smoothness.
The second order Taylor series locally approximates well, if satisfies some conditions, so the Newton’s method converges rapidly near . However, if is far from , the Newton’s method loses control of . In contrast, -smoothness guarantees a global upper bound so gradient descent iterations (with a fixed step size ) always converge, although it does not converge as fast as Newton’s method in the neighbourhood of .

Note that is a matrix if is a function mapping to . To describe the -Lipschitz of , we need to define a norm of matrices.
A simple idea is to view it as a -dimensional vector, and apply -norms. If , such a norm is called the Frobenius norm.

However, a more natural way is to consider the following definition. We may view an matrix as a linear map from to . So we can define its operator norm as follows.

Unless specified in context, we use to denote the spectral norm. Why do we call it "spectral norm"?

The spectrum of a matrix is the set of all its eigenvalues. This proposition shows that why this norm is called the "spectral" norm.

The advantage to use operator norm is that we usually need the Cauchy–Schwarz inequality, which is trivially true (by definition) under the operator norm: for all and , it holds that

Unfortunately, Newton’s method does not guarantee descent of the function values even when the Hessian matrix is positive definite. Similar to the gradient descent with a step size , we can modify the Newton’s method to include a small step size instead of , where the step size is chosen by a certain line search. This is called the damped Newton’s method.

Since is a descending direction (by convexity), we claim that there exists such that with parameter . Again, applying the backtracking line search, we can find such by starting from an initial (usually the initial step size is ) and repeating until the above condition is satisfied.

The convergence of the damped Newton’s method has two phase: damped Newton phase and quadratically convergent phase. We can show that there exists and such that the following holds. Specifically, assuming and for a constant satisfying for any , we have

Intuitively, the control of is related to the third derivatives of . For simplicity, we consider a univariate function . If is bounded then is Lipschitz. Previous analysis involves the bound of and separately. We will introduce another assumption of functions, in which we take into consideration both and simultaneously.

Furthermore, Newton’s method is affinely invariant. This means that the sequence of iterates remains unchanged under any affine transformation of the coordinate system. For example, consider . The Newton’s method gives Now we change the coordinates and let . The same function has the expression in the new coordinate system. Then the Newton’s method for gives If we start from the same point , then every iterate keeps the relation , namely, they are exactly the same point in two coordinate systems.

In general, suppose is nonsingular, and define . If we use Newton’s method (with the same backtracking parameters) to minimize , starting from , then we have for all , since and . However, the previous convergence analysis is not affinely independent. In contrast, the following assumption does not depend on affine changes of coordinates.

We now give some insights of this assumption. We stated that the gradient descent is the steepest descent method in the Euclidean norm, namely, the descent direction (after normalizing as ) is the solution to Now, in Newton’s method, if we define the local norm at as we can also see that the Newton’s method is the steepest descent in terms of the local norm. Namely, the descent direction with its normalizing vector is the solution to In fact, using the notation for simplicity, this optimization problem is equivalent to Applying Cauchy–Schwarz inequality, we have where the equality holds if for some constant , which gives the Newton iteration step. Now the convergence condition should be written using the local norm. The Newton local rule states that if the Hessian matrices satisfy that for all where , the Newton method gives quadratic convergence. Roughly, if is a univariate function, the Newton local rule says that that is The self-concordant functions include many of the logarithmic barrier functions that play an important role in barrier method and interior point method for solving convex optimization problems. In fact, the coefficient in the definition is not necessary, and it can be replaced by any constant . The standard choice is to guarantee that is a self-concordant function.

For strictly convex self-concordant function, we obtain bounds in terms of the Newton decrement It’s easy to see that in the gradient descent, we can use to measure the amount of decrement (e.g, the descent lemma). Now in Newton’s method, we should use the local norm instead of the Euclidean norm. Thus we define as the decrement.
There exists constants and (only depending on the backtracking line search parameters and ) such that the following holds.