We now consider equality constrained optimization problems. In a more compact form, it can be expressed as where and .
The feasible set of this problem is Recall that, such a set is an affine set. Given any , we find that Conversely, if , then satisfies and thus is in . So is the kernel (or the null space) of . Or equivalently,
14.2 Lagrange multiplier method for convex optimization
Question
How to verify that a solution is optimal?
We may rewrite the optimization problem as where is the feasible set.
Recall that, in Lecture 2, we’ve showed that if is a local minimum point of , then for any feasible direction (i.e. such that for any ), If is a convex function, is a convex set. Thus for any , is a feasible direction. Then we have for any . Conversely, if , then by convexity we have which implies that is a global minimum point. Therefore, we have the following optimality theorem.
Theorem
Suppose is a convex function. Then is a global minimum point of iff In particular, if or is open, then is a global minimum point iff .
For unconstrained optimization problems, we often assume the domain , or is an open set. So we can apply the criterion . Now, since we have constraints, the domain should be the feasible set , which is probably not open.
However, the feasible set is an affine set, which means that if , then . Thus, for all is equivalent to .
Note that for all , (or in a compact form, , where ). So if there exists such that (namely, , clearly we have for all .
Conversely, if for all , it holds that is a linear combination of (or equivalently, there exists such that ). There are multiple ways to prove this proposition. Suppose for the sake of contradiction is not a linear combination of . Then consider the orthogonal decompositionwhere and . By definition, . But which contradicts to our assumption for all .
An alternate proof is to use the rank-nullity theorem: for .
Lemma
Given a matrix , is the orthogonal complement of , denoted by .
Proof
We argue that is a subspace with the same dimension, thus . Note that for all , there exists such that . For all , . So which gives that . Therefore .
Moreover, we have by the rank-nullity theorem, which concludes that .
By this lemma we know that since .
The third way is to apply the Farkas’ lemma.
Theorem (Farkas' lemma)
Let . Then exactly one of the following sets is empty:
;
.
Recall that our goal is to show that where for all . Let . Then is equivalent to Applying Farkas’ lemma on and , we know that item is not satisfiable, since for all , if , then . Thus, there exists such that and . Let where we have .
Overall, we obtain the following Lagrange condition. The method to find the global minimum point by this condition is called the Lagrange multiplier method.
Theorem (Lagrange condition for convex optimization problems)
Consider a convex optimization problem Then is a global minimum point iff and there exists such that or equivalently, where and .
The constants are called Lagrange multipliers.
Example
The optimal solution is , where . The Lagrange condition has a geometric interpretation: the level set of should be tangent to the feasible set at the minimum point.
Here is an example in high dimensions.
Example
The optimal solution is , , , where the level set is tangent to the feasible set.
14.3 Lagrange multiplier method for general cases
It seems that the geometric observation (the level set and the feasible set should be tangent) still holds in general cases. We now introduce some examples.
Example
Both examples look good! However, note that in Example 1, there exists two parallel lines tangent to the level set .
So we know that the geometric observation (the level set and the feasible set should be tangent) is no longer a sufficient condition for minimum points. In fact, it is even not a sufficient condition for optimality. Consider the following example Two curves and are tangent at . Unfortunately, this is neither a minimum point nor a maximum point.
We may guess that the condition is necessary for optimality, although it is not sufficient. However, consider the following example The optimal solution is , where two curves are not tangent.
So what can we say about the minimum points? We first need to clarify the concept of “tangency”, and then we try to give a necessary condition for optimality and rule out the final example. We now introduce the following Lagrange condition for general cases.
Definition (Regular point and critical point)
Suppose is a function where . A point is a regular point of if has full (row) rank . Or equivalently, are linearly independent.
Otherwise, is called a critical point.
Theorem (Lagrange condition for general optimization problems)
Consider a general optimization problem If is a local minimum in the feasible set, and is a regular point of , then there exists such that The constants are called Lagrange multipliers.
Example
We can give an alternate proof of the Rayleigh quotient by Lagrange condition. Given a symmetric , consider the following optimizations This problem is to find extreme values of a continuous function on a compact set. So both maximum and minimum exist.
For a maximum (or minimum) solution , clearly is regular. By Lagrange multiplier condition, there exists such that . Namely, It implies that is an eigenvector of . Thus,
14.4 Tangent space and normal space
Recall the definition of “tangent lines”. Given a differentiable curve in , its tangent line at is . In other words, a tangent line is a line passing through with slope . We now generalize this concept to .
Definition (Tangent vector and tangent space)
Given a differentiable curve , the tangent vector of at is .
Given a differentiable surface (differentiable submanifold) , its tangent space at some , denoted by , is the set of tangent vectors at of all differentiable curves on and passing through . Namely,
An important (but probably surprising) fact is that the tangent space is a linear space. As an example, we show that the tangent space of is . Fix a point . For any differentiable where , we have (An alternate proof is by considering the derivative of , and noting that .) Thus it holds that . Conversely, for any , the curve has derivative . So we have .
The normal space is the orthogonal complement of the tangent space.
Definition (Normal vector and normal space)
A vector is a normal vector of at , if is orthogonal to the tangent space , i.e., for all .
The normal space of at , denoted by , is the set of all normal vectors at . Namely, is the orthogonal complement.
The key fact is that, for any optimization problem, is a normal vector of the feasible set at , if is a minimum point.
Lemma
For any optimization problem with objective function and feasible set , if is a local minimum of over , then (or equivalently, ).
Proof
By definition, for every , there exists such that and . Note that is a local minimum of over , so is a local minimum of over . Thus we have . Since we conclude that and hence .
The remaining ingredient is how to represent . Suppose the optimization problem has constraints . Using a similar argument we have since is a constant function. Let . Then For any , we have (or equivalently, ). Finally, we show that if is regular, then . Combining with , it follows that .
Lemma
If is a regular point, then for any , there exists a differentiable curve such that and .
The proof is an application of the implicit function theorem, and we omit the details here.
Example
Consider the following examples andNote that they have the same feasible set . So the optimal solutions are the same (). The tangent space of the feasible set at the optimal solution is . We can verify that is orthogonal to the tangent space.
However, in the first example, the gradients of constraints at are and , which yields that is not regular and Lagrange multipliers do not exist. In contrast, in the second example, the gradients of constraints are and , whose linear span is exactly the normal space at . So the Lagrange condition applies.
14.5 Second order condition
It is not difficult to write down a second-order condition. Again, note that if is a local minimum of over , then for any with , is a local minimum of . Applying the second-order condition for local minima, we have , where Similarly, we also have for every . By the Lagrange condition, there exists multipliers such that so it follows that Overall, we have the second-order Lagrange condition.
Theorem (Lagrange condition for general optimization problems, second-order)
Consider a general optimization problem If is a local minimum in the feasible set, and is a regular point of , then there exists such that and for all ,
