lec16

Lecture 16. Karush-Kuhn-Tucker Conditions

### 16.1. Problem formulation and active constraints

We consider the constrained optimization problem

Its feasible set is

We would like to establish necessary and sufficient (in some sense) conditions for optimality. The first observation is that constraints have different effects for different points.

Definition (Active and inactive constraints)

Let . An inequality constraint is called active at if

It is called inactive at if

The active index set is

Only active inequality constraints can affect the first-order geometry of the feasible set at . Inactive constraints are locally irrelevant: if and is continuous, then in a sufficiently small neighborhood of . Thus in a neighborhood of , the feasible set remains the same locally if we ignore .

16.2. KKT conditions and geometric meaning of the multiplier sign

Suppose is a local minimum. Then is also a local minimum of the equality-constrained problem obtained by keeping all equality constraints and turning only the active inequalities into equalities:

If is a regular of the above equality-constrained problem, the Lagrange multiplier theorem gives multipliers and , , such that

For every inactive constraint , set . Then we have

and

The second condition is called complementary slackness. We also have the genuinely new condition for inequalities, which is the sign condition

But why we have the sign condition? Consider the following example:

Let

The Lagrange condition is

if the problem is restricted to .
Here

So we have two solutions and . Both solutions are optimal (maximal / minimal) for the equality-constrained problem. But for the minimization problem, we want to rule out the maximizer.
At , stationarity requires

so , which violates .

At , stationarity requires

so , which satisfies .

Thus the nonnegative multiplier condition rules out and keeps the true minimizer . Geometrically, for a constraint written as , the vector points toward the outside of the feasible region. At a constrained minimum, the objective gradient should point toward the inside of the feasible set, which must be balanced by a nonnegative combination of outward constraint gradients.
Pasted image 20231209094520.png
Now we can present KKT conditions, which is necessary for regular points. According to the discussion above, here regular means regular for equality and active inequality constraints.

Definition (Regular point / LICQ)

A feasible point is called a regular point if the gradients

are linearly independent. This condition is also called the linear independence constraint qualification (LICQ).
A point is called critical if it is not regular.

Theorem (Karush-Kuhn-Tucker necessary conditions)

Consider

Assume that are continuously differentiable. If is a local minimum and is a regular point, then there exist multipliers

such that:

Stationarity:
Primal feasibility:
Dual feasibility:
Complementary slackness:

The multipliers and are called Lagrange multipliers or KKT multipliers.

Define the Lagrangian

where

Then the KKT conditions can be written as

with

Equivalently,

and

16.3. KKT sufficiency for convex optimization

Similar to the Lagrange conditions for equality-constrained problems, for general nonlinear optimization problems, the KKT conditions are necessary under a constraint qualification, but they are not sufficient. But for convex optimization problems, they are also sufficient.

Theorem (KKT sufficiency for convex problems)

Consider

Assume that and are convex and differentiable, and that each is affine. If is feasible and there exist multipliers and satisfying the KKT conditions, then is a global minimizer.

Proof

Let be any feasible point. Since is convex,

Thus it is enough to prove

By stationarity,

Since each is affine and both and are feasible,

For inactive constraints, complementary slackness gives . For active constraints, convexity gives

Since and ,

Together with , this implies

Therefore , and hence for every feasible .

16.4. Examples

Example 1

Consider

This is a convex problem, so KKT conditions are sufficient.
In addition, there is no critical points, so all optimal solutions must satisfy KKT conditions.

The Lagrangian is

where . The KKT conditions are

From the first three equations,

If , the inequality is inactive and , so . If , the same point satisfies KKT with . If , the inequality must be active, so

Therefore

Example 2

Consider

This is a convex problem, so KKT conditions are sufficient.
In addition, there is no critical points, so all optimal solutions must satisfy KKT conditions.

The KKT conditions are

We examine the active sets.

If both constraints are inactive, then and , which is infeasible.
If is inactive and is active, then and . Solving gives and , but this violates .
If is active and is inactive, then and . The stationarity equations imply and , so , which is infeasible.
If both constraints are active, then Thus or . At , stationarity gives so and . At , stationarity gives which violates dual feasibility.

Therefore the optimal solution is

Pasted image 20231209095749.png

Example 3 (Linear programming and complementary slackness)

Consider

Write the nonnegativity constraint as . The Lagrangian is

where and . The KKT conditions are

The corresponding dual problem is

If and are primal and dual optimal solutions, complementary slackness says

The KKT multipliers are

Thus the KKT conditions for linear programming are exactly primal feasibility, dual feasibility, and complementary slackness.

Remark

We should notice that it is possible that a convex problem has optimal solutions but KKT condition is unsolved, because KKT is only necessary for regular points, but critical optimal points may exist.

Example 4

Consider the following problem: The KKT condition is This is unsolved, since the only feasible point is , which violates the KKT conditions.
But is still optimal, since it is actually a critical point.

16.5. Proof of the necessary conditions

Let be a local minimum and let be its active index set. Since inactive constraints are strict at , continuity implies that there is a neighborhood of in which all inactive constraints remain feasible.

Therefore, locally, the feasible set is determined by

and

Because is regular, the equality-constrained Lagrange multiplier theorem applies. Hence there exist and for such that

Set for . Then stationarity and complementary slackness follow.

It remains to prove for active constraints. Suppose, for contradiction, that there exists such that

Consider the manifold defined by all equality constraints and all active constraints except :

Its tangent space at is

By regularity, is not in the span of the other active constraint gradients and equality constraint gradients. Hence there exists such that

Replacing by if necessary, assume

Multiplying stationarity by gives

Since and , we get

By the regularity of , there exists a differentiable curve

such that

Then

and

Thus, for all sufficiently small ,

The curve satisfies all equality constraints and all active constraints except possibly . The constraint becomes strictly feasible for small , and all inactive constraints remain feasible by continuity. Hence is feasible and has a smaller objective value than , contradicting local optimality.

Therefore for every . Since inactive constraints have , dual feasibility holds for all .