Recall that the gradient descent converges slowly if the condition number is large. For example, consider the function . At some point , . It locally decreases rapidly but not globally. The ideal descending direction is , which can also be written as In general, if where , the ideal direction at is since .

More generally, recall the Newton's method introduced before for finding roots, where we use a Taylor series to estimate the objective function. When is not quadratic, consider its Taylor approximation Denote by the right hand side. Note that if is invertible. Thus the minimizer of is . Then, we let be the minimizer of the second-order Taylor series at . This method is called the Newton's method for optimization.

Note that if the objective function is strictly convex, then , which implies that is invertible and . This is because if 's eigenvalues are all positive, their inverse are also positive. Therefore, .

Recall our requirement for the descending direction. We hope . Clearly, if , is a descending direction, since unless .

Intuitively this is not true since we use the second order Taylor series to approximate the function, but the Taylor series only works locally.

If the second-order Taylor series estimates the value of functions well, then given by the Newton's method is the minimum point of the Taylor series and it should be close to the minimum point of . But what happens if the Taylor series doesn't approximates well? Now we consider some "bad" examples.

The reason why the Newton's method does not converge in this example is that the second-order derivative of is close to zero when is large, which yields that the second-order Taylor series is not a good approximation near the minimum point. The Taylor series approximates well in the neighborhood of , but loses control at , if is large.

However, just keeping small is not enough yet. Here is another "bad" example.

In this example, the reason to failure is that changes rapidly so the second order Taylor series cannot approximates well even in the neighborhood of .

We now give some conditions to guarantee the convergence of the Newton’s method iterates.

Recall the gradient descent iteration , where we actually calculate the minimum point of the following function and let be its minimizer to approximate the minimum point of . In particular, if , is an upper estimation of by -smoothness.
The second order Taylor series locally approximate well, if satisfies some conditions, so the Newton’s method converges rapidly near . However, if is far from , the Newton’s method loses control of . In contrast, -smoothness guarantees a global upper bound so gradient descent iterations (with a fixed step size ) always converge, although it does not converge as fast as Newton’s method in the neighbourhood of .

Unfortunately, Newton’s method does not guarantee descent of the function values even when the Hessian matrix is positive definite. Similar to the gradient descent with a step size , we can modify the Newton’s method to include a small step size instead of , where the step size is chosen by a certain line search. This is called the damped Newton’s method.

Since is a descending direction (by convexity), we claim that there exists such that with parameter . Again, applying the backtracking line search, we can find such by starting from an initial (usually ) and repeating until the above sufficient decrease condition is satisfied.

Another way to control is to compute the third derivative. For simplicity, we consider a univariate function . If is bounded then is Lipschitz. Previous analysis involves the bound of and separately. We now introduce another assumption of functions, in which we take into consideration both and simultaneously.

Moreover, Newton’s method is affinely invariant. Suppose is nonsingular, and define . If we use Newton’s method (with the same backtracking parameters) to minimize , starting from , then we have for all . However, the previous convergence analysis is not affinely independent. In contrast, the following assumption does not depend on affine changes of coordinates.

The self-concordant functions include many of the logarithmic barrier functions that play an important role in barrier method and interior point method for solving convex optimization problems.

In fact, the coefficient in the definition is not necessary, and it can be replaced by any constant . The standard choice is to guarantee that is a self-concordant function.