lec13

So far we only consider minimizing differentiable functions. In this lecture, we focus on how to solve the optimization problem for much more families of convex functions. We will generalize a method of gradient descent named the proximal gradient descent for nondifferentiable functions.

Recall the gradient descent iteration , which can be viewed as approximating and minimizing. We first approximate the function near :and then let be the minimum point of approximation function . Now we assume is not differentiable, but can be divided into two parts:where is convex and differentiable, and is convex but not necessarily differentiable. Then we define to approximate and let to approximate the minimum point of .

Note that where is a constant not depending on (assuming that and are fixed). So we obtain Here is the gradient descent iteration if we would like to optimize only . So roughly speaking, after adding , we hope locate near the local minimum of , and not make large.

Now we define the proximal operator as follows.

Then we can rewrite the proximal gradient descent method as

The key is that we need to decompose properly. Clearly we can set and . But this decomposition is meaningless since wo do not know how to compute the proximal operator at all.

Fortunately for some important problems we have a "good" decomposition. For example, we consider the problem of linear progression avoiding overfitting. Suppose we have a data set where and and we assume that for some .

Let We can compute coefficients by solving the following optimization problem (least square method) However, there are usually some redundant coefficients that may cause overfitting. So we hope to add some constraints, such as, the number of nonzero entries in is at most . Unfortunately this problem is not a convex optimization after adding this constraint (why?). We still need to approximate it. One idea is to use the following approximation where is a parameter. This method is called LASSO (least absolute shrinkage and selection operator).

The objective function has a clear decomposition: and . The advantage is that the proximal operator is easy to compute with respect to this .

Given , the proximal mapping is Note that in this optimization, all entries in are independent, so we can solve this optimization by solving the following optimization for each entry: where is called the soft thresholding operator.

Finally, recall that , so the iteration of the proximal gradient descent is This algorithm is called the ISTA (iterative soft-thresholding algorithm).

We now show the correctness and convergence of the proximal gradient descent.

Assume that is -smooth and set . We write the proximal gradient descent as the following form where We would like to show that performs like in the analysis of the algorithm. We should remark here by the view of backward Euler method, is actually (although does not exist). We will formally explain it later.

We first claim that is a descending direction. In fact, we always have . Since we have by -smoothness of , it yields that Next, we can show that if and only if is an optimal point of . A trivial example is . Then we have and thus . So if and only if is a minimizer of . The following theorem asserts the general cases.

The "only if" direction is easy. If , all inequalities are tight, so . However, because is convex and is strictly convex, is also strictly convex, and thus has a unique minimizer , which gives that .

Now we would like to show that if then . In other words, for all , . How can we show this? A naive idea is to show that and . However this idea looks too good to be true.

A reasonable method is to use convexity. Note that . So we hope This inequality looks like the first order condition for . Here performs like the gradient of . Since is not differentiable, we would introduce the notion of subgradients.

If is convex, then subgradients always exist (but may not unique). Just consider the supporting hyperplane of the epigraph of .

This lemma immediately implies that if , then . Thus, ,

Using this lemma, we can also obtain the descent lemma for the proximal gradient descent. Since , we have that is, (recall the backward Euler method). It gives that Applying the -smoothness of , and choosing , we have that Combining them together, it follows that Finally, we present the results of convergence rate.

Similar to the convergence analysis of gradient descent, we bound the difference again. Note that Assume that is -strongly convex (allowing here). Thus, we obtain that which further yields that The remaining part is the same as the analysis of the gradient descent method.