We now study the general convex optimization problems. First, we consider the easiest case: no constraints. Namely, the optimization problem is where is a convex function.

Recall that, the optimality condition for convex functions is

For convenience, we assume that , the objective function is differentiable and has a finite minimum point (and the minimum value ). For some simple cases, we can compute the minimum point by solving the equation . However, in general we cannot expect that closed-form solutions always exist. So we introduce some algorithms to find optimal solutions.

Analogously to the simplex method, we would like to move from a solution to a better “neighbor” . The convexity guarantees that As we hope is better, i.e., , it requires that . Conversely, we know that if the directional derivative , then there exists such that . So is a reasonable requirement for the moving direction .

This inspired the so-called descent method: start from a solution and move to iteratively, where is the step size to be determined and is the moving direction satisfying .

The first question is when we can stop? Of course the ideal stopping criterion is for some . If so, we know that is indeed a minimum point. However, in practice, we cannot expect this happens. So we usually use stopping criteria such as , , or iterations.

The next question is, does this algorithm converge to an optimal solution? In fact, we claim that if we assume that only depend on , and the choice of satisfies for every (note that the optimal solution may not be unique), then the value of objective functions generated by the descent method converge to the minimum value . (However, may not converge and we will give an example later.)

We assume has a finite minimum value , and . So has a limit as goes to infinity. Now we would like to show that the limit is .

Let . Intuitively, if , as we hope as long as , we can argue that still decreases too fast even if is sufficiently close to .

Rigorously, let . Then is a compact set, if we assume for convenience (otherwise may not be necessarily bounded). Let be a function defined by where are the step size and the direction we choose if . That is, measures the difference between and if we set .

By our assumption as long as , and noting that if , we conclude that for all . Applying the extreme value theorem, there exists which implies that for every .This contradicts our assumption that there exists such that , and thus completes the proof.

We now consider a specific descent method, the gradient descent, where we select . Then trivially .

There is an advantage to choose since it is the direction of steepest descent, namely, the value of decreases most rapidly: For any unit length vector , the directional derivative satisfies by the Cauchy-Schwarz inequality, and the equality holds iff .

Applying this choice of directions, we obtain the gradient descent method:

We now consider how to choose the step size . Intuitively, the choice of step size can effect the converge rate of the algorithm.

Let's start from an easy example: where . Since we hope , it requires that , which is equivalent to So suffices.

Next, consider the multivariate function where . Now . So It is sufficient to find a value of such that for all , . We need the following lemma.

Note that in this proof we do not really need . This lemma holds for all symmetric . Applying this lemma, it gives that suffices in the gradient descent method for quadratic functions.

However, for general cases, we cannot expect a universal condition for . For example, consider the function . If we choose to be a constant , no matter what value is, the algorithm does not work as long as .

We would like to avoid functions similar to , where changes too drastically near .

We usually use -norm, unless otherwise specified.

An -Lipschitz function is continuous, but may not be differentiable. Intuitively, for a Lipschitz continuous function, there exists a double cone (white) whose origin can be moved along the graph so that the whole graph always stays outside the double cone.

Recall that we hope does not change rapidly. So we define the following notion of "smoothness".

We use the notation if . Then we have the following equivalent definitions.

Note that if , we can easily prove the “” direction since the mean value theorem gives that for some . However, there is no such theorem for vector-valued functions.

An -smooth functions may be not convex. If is further convex, all absolute values are not necessary.

Recall that is convex iff , which shows that is underestimated by an affine function. Now, if is -smooth, it is overestimated by a quadratic function.

Recall that, we hope to find the value of the step size such that . Now we assume that is -smooth. Then if we set . In particular, if we choose , it gives the following descent lemma.