lec15


Lecture 15. Newton’s Method with Equality Constraints


15.1 Lagrangian function

Consider an optimization problem We can reformulate the Lagrange condition in a compact form. We define the Lagrangian function as The Lagrange condition shows that if is regular and local minimum, then there exists such that Given the definition of Lagrangian, we can rewrite it as Note that if is feasible. So we can simplify the Lagrange condition as .

Conversely, for convex optimization problems, the Lagrange condition (combining with the feasibility) is sufficient for optimal solutions, which is exactly and . Thus, we have the following sufficient and necessary condition for convex optimizations.

Theorem (Lagrange condition, expressed by Lagrangian)

For any point , it is optimal for the convex optimization problem if and only if there exists such that

Note that is not convex in general (see e.g., , , and thus ), although is a convex function of for any fixed . So does not imply is a minimum point of .
In fact, it is a saddle point in a sense. For feasible , i.e., , it is easy to see that . For infeasible , there exists , so we have . Therefore, we conclude that Moreover, for convex optimization problems, the other direction also holds, namely, we have the following proposition.

Theorem

For the Lagrangian of a convex optimization problem, if , then

Proof

For the first equality, note that if , . Thus .
Now we define . Our goal is to show that .
We first show that there exists such that . In particular, we show that . Note that . Since is fixed, is a convex function of . By the first order condition for convexity, achieves the minimum value if and only if . Since , we have .
Next, we show that for all , . Denote by the feasible set . Then for all and all , . So for all ,

Later, we will revisit this theorem in the context of general convex optimization with inequality constraints.

15.2 Karush-Kuhn-Tucker System

We now consider how to solve convex optimization problems with equality constraints. Recall the problems with no constraints, where we approximate the objective function by a quadratic function and minimize the quadratic function. We would like to apply the same idea. So the first step is to minimize quadratic functions with equality constraints.

Given , , and , consider the following quadratic problem with equality constrains We can solve this problem by the Lagrange multiplier method. The Lagrangian function is and the Lagrange condition is We know that is a global minimum if and only is a solution to the above system of equations for some . The above system of equations can be expressed in the following matrix form: which is called the KKT system and the coefficient matrix is called the KKT matrix.

Block Gaussian elimination

If we further assume is invertible (nonsingular), we can solve the KKT system by Gaussian elimination. Because , it is equivalent to .

Left multiplying to the first row and subtracting the second row, we obtain that So Plugging it into the first row of the original KKT system we have

Remark

Both and must exist in the above calculation. If we assume and ( has full row rank), then both of them exist.
First, implies that is invertible. Next, if is not invertible, there exists such that . If we left multiple to both sides, we can get Since , its rows are linear independent. So , and contradicts to the fact that .
The assumption of is reasonable. Otherwise either the problem is infeasible (e.g., constraints are and ), or there are redundant constraints (e.g., constraints are and ).
But the assumption of is strong. Actually, we only need the KKT matrix to be invertible.

Nonsingularity of KKT matrices

Let where . If there is no constraints, when does attains it minimum value? The gradient is . So there exists an optimal solution if and only if the equation has solutions, namely, is in the row space of . Recall the lemma . It is further equivalent to , i.e., There are two cases:

  1. if is invertible / nonsingular, there exists a unique solution ;
  2. if is not invertible, there are infinite many solutions.

Now let be the KKT matrix. Clearly the KKT system is solvable if and only if .

We first prove the following useful proposition.

Lemma

Suppose , then for any vector , if and only if .

Proof
  • Necessity: Trivial.
  • Sufficiency: Since , we can do eigendecomposition to . where is the -th eigenvalue and is the -th eigenvector of . Then we have which means either or . Hence,

Now we can show that . Note that if , then Conversely, we claim that if then and . This linear equation system is equivalent to and . Note that by left multiplying , we obtain thatSo by the above lemma. Moreover, , contradicts if .

Then we focus on the case where is invertible / nonsingular.

Theorem

KKT matrix is invertible (nonsingular) is equivalent to each one of the followings:

  1. , or
  2. if and , then , or
  3. , if , then .
Proof
  • "invertible 1". If there exists such that , we have which means the KKT matrix is not invertible.
  • "1 2". If and , then . So . Thus by the above lemma.
  • "2 invertible". By the above lemma, item 2 is equivalent to if and then . So . Thus , which implies that is invertible.
  • "2 3". If , for all , . Note that . So . Thus by linear independence. If then . So .
  • "3 2". If , for all that , there exists such that . Since , .

15.3 Newton’s method

Now we consider how to solve general convex optimization with equality constraints.

Recall Newton's method. Given , we do Taylor Expansion of at :

  • If there is no constraints, we use to approximate the minimum point of .
  • If there are constraints , we may use to approximate the minimum point of under constraints .

Let . We have and the constraints become since If is a feasible solution, we have , hence .
Now for , the problem becomes and finally The KKT system for optimizing is

Note that we use as stopping criteria instead of . This is because, when the algorithm should terminate, its gradient is not zero (it needs to follow the Lagrange condition).

Next, let us see some examples of using Newton's method.

Example 1

Start from . The KKT system at this point is Its solution is . The next point is exactly the optimal solution .

Example 2

Start from . The KKT system at this point is Its solution is . The next point is also exactly the optimal solution .
In this example, although is not invertible, the KKT matrix is still invertible.
We may check by the criterion introduced above, since and , which satisfy the previous condition of .

Example 3

Start from . The KKT system at this point is Its solution is . The next point is . It is not the optimal solution, but it is also in the right direction.

15.4 Correctness and convergence

We will demonstrate the correctness of Newton's method via the following two propositions:

  1. Each time we choose a descending direction: .
  2. The stopping criteria is correct: if then is optimal.
Proposition 1

unless .

Proof

The KKT system can be unfolded as follows: Use to left multiply the first line, we can get which yields that (unless )

Proposition 2

iff is optimal.

Proof

First, (due to the lemma proved in Section 15.2). Then by the first equality in KKT system, , so

Now we analyzed the convergence of Newton's method with equality constraints. In fact, we can convert equality constrained problems to problems without constraint. For example, the following problem is equivalent to .
In general, for an equality-constrained problem, assume its feasible set is . It can be rewritten as for some and , since is an affine set.
Then the original problem is equivalent to the following one: Next we would like to show that iterations are identical if we apply Newton’s method to these two optimization problem.

Applying Newton's method to , we will get In fact, as we will see later, satisfies . Note that , so (and thus invertible) if and only if the KKT matrix in the Newton’s method to optimize is invertible.

Assuming , we have the following proposition, which reveals that the Newton’s method is affinely invariant. If is invertible, then . Thus the proposition is trivially true. But in general cannot be invertible, so we should be careful.

Lemma

For all ,

Proof

By induction, assume . Let and .
For all , if then , which implies , and vice versa. So is exactly .
Note that by Newton's method. Thus there exists such that .
Moreover, by the first equation of KKT system, we know that Left multiplying on both sides, it follows that Since is , we have for all , which implies that . Then, we have which implies that (noting that iff KKT matrix is invertible) Hence, and

Therefore, the convergence of can lead to the convergence of .

Example 4

Start from . The next point is .
Now we optimize another form of this problem. It is easy to see that this problem is equivalent to The relation between and is So we start from to optimize . Using Newton’s method, the next point is keeping the relation between iterations of and .

Example 5

The KKT system is Start from . Then the KKT system at this point is Solving this system of equations we obtain , and the next point is .
Now we optimize another form of this problem. It is easy to see that this problem is equivalent to The relation between and is So we start from to optimize . Using Newton’s method, we have and thus the next point is keeping the relation between iterations of and .