19.1 Lagrange dual function and Lagrange dual problem
Recall the LP dual problem, we can find the lower (upper) bounds of linear programs from their constraints. Similarly, we would like to find lower bounds of general optimization problems. For example, consider the following problem We can assign a coefficient to the constraint and hope can somehow be a lower bound of the objective function. To this end, assuming , we need to find a constant such that holds for all , which gives . In addition, we hope be as large as possible. Thus we have For each , gives a lower bound of the objective function. For example, if , then ; if , then . Again, we would like to find the largest lower bound of the objective function, which is the maximum value of . This is another optimization problem, and is called the Lagrange dual problem of the primal problem. In general, suppose we have a primal optimization problem For each coefficients , and , let It always gives a lower bound of the primal problem. The dual problem is to find the largest lower bound, that is, , where and . Recall that, if we define the Lagrangian as follows where and , then the domain of is given by and can be expressed as .
Another natural viewpoint is from Lagrange multiplier condition and KKT conditions. Note that for equality constrained problems, if is an optimal solution and Lagrange multipliers exists, is a saddle point of , since Lagrange multiplier condition gives . For inequality constrained optimization problems, the situation becomes more complicated. The KKT condition can be expressed as for some KKT multipliers and . Since may not be in the KKT condition, is not a stationary point any more. But it is still a minimax point. Note that if is feasible, then for all and for all . So we have for all and . Moreover, we have . Thus we obtainFor infeasible , there exists or . Then we know as or goes to infinity. Overall, if is a minimum point of over the domain , we conclude that This equality holds even when we consider general (not necessarily convex) optimization problems. Recall that in LP duality, a corollary of the strong duality is , where the minimization and the maximization can be exchanged. However, what happens if we exchange the order in general optimization problems? We obtain the following optimization problems, which are called dual problems, and still give lower bounds for primal problems.
Definition (Lagrange dual)
Given an optimization problem and its Lagragian function , the Lagrange dual function is defined by and the Lagrange dual problem is given by Sometimes we write , instead of , for convenience.
We first introduce some examples.
Example 1
Primal optimization problem: Lagrangian function: Dual function: Dual problem: The optimal solution is with optimal value .
Example 2
Primal optimization problem:Lagrangian function: Dual function: Dual problem: Let , then the optimal value is .
Example 3
Primal optimization problem: Dual function: Dual problem: , which is equivalent to
The Lagrange dual problem of a linear program is exactly the same as the dual problem introduced before.
An interesting fact is that dual functions are always concave, thus dual problems are always convex, regardless of whether the primal problem is convex or not.
Theorem
For any (not necessarily convex) optimization problem, its Lagrange dual function is concave.
Proof
For any fixed , is an affine function of and , so is a pointwise minimum of a family of affine functions, which implies that is a convex function.
19.2 Weak and strong duality
What is the relationship between the optimal value of the primal and the optimal value of the dual?
Let be the domain set, be the optimal value of the primal problem, and be the optimal value of the dual problem. Then we have the following weak duality theorem.
Theorem (Weak duality)
Proof
As we mentioned before, For any fixed , we have Thus,
In fact, the weak duality theorem is a particular case of the following min-max inequality.
Theorem (Min-max inequality)
Let be two sets. For , it holds that
The min-max inequality has many varieties, such as the limit superior and the limit inferior of a sequence. Given a sequence , we have , where and Now we define the duality gap as , which is always nonnegative. If , then we say strong duality holds.
Unlike the duality for linear programs, strong duality does not always hold. See e.g., Example 4 above, where and . A natural question is, under which condition the strong duality holds?
Our first result is that for convex optimization problems, KKT condition implies strong duality.
Theorem
For any convex optimization problem, if a feasible solution has KKT multipliers and , then strong duality holds. In particular, is an optimal solution to the dual problem.
Proof
For convex problems, if has KKT multipliers, then . By weak duality, we have So it suffices to show that .
Note that since and . We fix and let . Then . Because is a convex function of and , we conclude that . Thus,
Remark
From this proof, we find that for , if and only if is an optimal solution to the primal problem, is an optimal solution to the dual problem, and strong duality holds, because This fact holds for any optimization problem (not necessarily convex).
Conversely, if the primal and the dual have finite optimal solutions, and strong duality holds, then KKT condition is satisfied.
Theorem
For any optimization problem, if the primal has a (finite) optimal solution , the dual has a (finite) optimal solution , and the strong duality holds (or equivalently there exists such that ), then are KKT multipliers of .
Proof
Note that is feasible, so for all , and for all . Thus, The first inequality is due to . By strong duality, It yields that We conclude that KKT conditions are satisfied, since
;
for all ;
is feasible for the dual .
Here, the existence of finite optimal solutions is essential. Only strong duality is not sufficient. Otherwise, see e.g., Example 2 above, where does not have KKT multipliers.
Tip (KKT conditions revisit)
In a sense, KKT conditions are equivalent to existence of optimal solutions and strong duality. So we can rewrite KKT conditions in terms of the primal (P) and dual (D) problems:
Stationary: ;
Primal feasible: is feasible for (P), namely, for all and for all ;
Dual feasible: is feasible for (D), namely, for all ;
Complementary slackness: , namely, for all .
Sometimes we may find solving KKT conditions is difficult. Is there any simple condition to certify strong duality?
Geometric interpretation of duality
Before introducing other conditions, we first give a geometric interpretation for strong duality.
Example
Consider the following optimization To analyze this problem, we can draw an “epigraph” of with respect to . Namely, let Then is the point where intersects -axis.
What is ? The Lagrangian is Given a fixed and a constant , characterizes a line intersecting -axis at with slope . So by letting , we see that is the lowest intersection between -axis and a line which intersects the boundary of and has slope . Namely, is the intersection between -axis and the supporting lines to with slope . So is the highest intersection between -axis and all supporting lines to .
19.3 Slater’s condition for convex optimization
Now we give the Slater’s condition for convex optimizations.
Theorem (Slater’s condition)
If there exists such that for all and for all , then strong duality holds.
Remark
Here the means relative interior points. A point is a relative interior point if , where is the affine hull of . For any convex set , the relative interior is equivalent to the set of non-extreme points. We can easily see the differences between interior and relative interior. For example,
the interior of a line segment in an at least two-dimensional space is empty, but its relative interior is the line segment without its endpoints;
the interior of a disc in an at least three-dimensional space is empty, but its relative interior is the same disc without its circular edge.
Before proving this theorem, we first ask when strong duality does not hold. The geometric interpretation tells us is the lowest intersection of and -axis, while is the highest intersection of -axis and all supporting hyperplanes of . So is a point on the boundary of . Heuristically, if is convex, there should be a supporting hyperplane passing through , which gives that .
Unfortunately, this observation is not always true, because there are vertical (orthogonal to -axis) supporting hyperplanes. Consider the following example (Example above).
Example
The dual function is if and otherwise. So the optimal value to the dual problem is , while the optimal value to the primal is . Strong duality does not hold.
Why does strong duality fail in this case? We can also define We can find that
if , then , so ;
if , then , so as .
Thus, . Although is convex, strong duality does not hold, since the unique supporting hyperplane at is vertical.
So geometrically, the Slater’s condition says that if is convex, and in the region is not empty, then strong duality holds.
We now prove the Slater’s condition. We first show that is convex. Generally, for a convex problem we define
Lemma
is a convex set.
Proof
Take two points , so there exists such that For any , let . Then by convexity of and affinity of , we have This leads to .
Now we are ready to prove strong duality if Slater’s condition is satisfied.
Proof of strong duality
If , by weak duality, , so . Thus the strong duality holds.
Now assume that . By Slater's condition, the feasible set , so . It suffices to show that there exists a nonvertical supporting hyperplane passing through .
Step 1. Prove that . Note that . So for all , there exists such that Thus . Moreover, for all , there does not exists such that which gives that . Therefore, .
Step 2. Show that there exists a nonvertical supporting hyperplane. Since , there exists a supporting hyperplane passing through , i.e., there exists such that Note that for all , is in . So we have . Similarly, for all , is in by definition, so we have .
We now claim that . If not, . By Slater's condition, there exists such that Hence there exists such that Thus , where . Since , it implies that . Now we obtain that for all , where . We can show that it is impossible unless .
Suppose for some and . Without loss of generality, we may assume the matrix has full (row) rank. Otherwise some constraints are redundant on the domain . Note that is a relative interior point of . If there exists and such that , then there exists such that . Since is affine, and , we have If for all , we can conclude that for all . Namely, for all , On the other hand, , so for every . Using the same argument for the sufficiency of Lagrange multipliers for convex optimization, we conclude that is in the row space of . But this is impossible unless , due to the full row rank assumption of .
Overall, we obtain that if , then and , which contradicts to . It yields that , and further gives that .
Step 3. Conclude that strong duality holds. Now let , . For all , since , we have Thus, because for all . By weak duality , we conclude that .
19.4 An example of solving optimization via duality
Now we show an example of solving optimization problems via duality. Consider the problem finding the projection to the polytope. We compute its Lagrangian function and let . By Slater's condition, to solve the primal optimization problem, it suffices to solve its dual problem Observe that, is a strongly convex function for . Then for , the minimizer of is attained at the point . So it holds that where To (approximately) maximize we can apply the projected gradient descent.
