lec17


Lecture 17. Projected Gradient Descent


17.1 Projection operator and projected gradient descent

To solve the inequality constrained problems, we introduce the projected gradient descent.

Recall the iteration step in the gradient descent method, . Now we need to minimize over a feasible set . If is feasible, then we can run the gradient descent iteration. If is infeasible, a simple idea is to project it onto . This method is called the projected gradient descent.

Definition (Projection)

The projection of a point onto a set is the point in the set with minimum distance to the given point. Namely, the projection operator is defined by

The the projected gradient descent step can be given by Let the iteration step can be expressed as Recall that, in Lecture 4, we show the following lemma.

Lemma

Let be a nonempty, closed and convex set. Given and , for any , it holds that .
1696152640351.png

Conversely, if there exists such that , we have . Otherwise, let . Then we have However, we also have , which implies that if . Contradiction.
Thus, if and only if for any .

Applying this lemma, we can show that plays a similar role as in the gradient descent.

Lemma

For any , The inequality holds if and only if .

Proof

Since , we have which gives that Thus,

So we know that is a desceding direction. Now we show that if then is a minimum point.

Lemma

is a minimum point of over , iff , namely, .

Proof

Applying the above lemma, we have if and only if for all , which is further equivalent to We conclude this lemma by the first-order optimality conditions of convex functions.

Hence, in the projected gradient descent, we can stop when is small, or equivalently when is small.

17.2 Examples of projection operator

Projected gradient descent is useful when the projection operator can be computed efficiently. Here we give some examples.

Example 1 (Box constraints)

Pasted image 20231210235303.png
It is easy to see that

Example 2 ( constraints, ridge regression)

Pasted image 20231210235926.png
The projection operator is to compute By KKT condition, there exists such that Then we have .
Hence, .

Example 3 ( constraints, LASSO)

Pasted image 20231211000252.png
Unfortunately, there is no closed form for the projection operator . But we can compute it efficiently.
By symmetry, we only need to consider the case where for all . Now is equivalent to the following optimization problem: By KKT condition, assume there exist KKT multipliers such that

  • Case 1. , then . Hence .

  • Case 2. , then , hence . Since , we have .

    • If , by , we have .
    • If , by , we have .

    Now we have and .
    We may use the binary search to find , where the lower bound is and the upper bound is .

17.3 Comparison with proximal gradient descent

To analyze the convergence of the projected gradient descent, we show that it is a special case of the proximal gradient descent.

Let be the indicator function of , defined by Clearly is a convex function if and only if is a convex set.

Then we can show that the proximal operator for is simply the projection onto : Since and for any , we find that the projected gradient descent for is the same as proximal gradient descent for .
By extending the results on Lecture 13 of to , the convergence analysis for proximal gradient descent applies also to projected gradient descent.

Theorem

Let be a nonempty convex set, and be an -smooth convex function over . Suppose is a minimum of over . Then the sequence produced by projected gradient descent with constant step size satisfies and Furthermore, if is also -strongly convex, then