lec04

Lecture 4. Supporting and Separating Hyperplane Theorem

We would like to present a fundamental property of convex sets. Roughly speaking, we would like to show that every convex set can be characterized by its ‘supporting hyperplanes’, and every two convex sets can be separated by a hyperplane.

4.1 Projection to convex sets

Given a set , the distance between a point and is defined by We define the (metric) projection of onto as the closest points in to .

Definition (Projection)

Let be a nonempty, closed and convex set. Then for any , the projection of onto is defined as That is, .

Question

Is this well-defined?

Clearly, if then . Now we assume that .
We first show that the minimizer exists. Since , select any and let . Then . Since is closed, is bounded and closed, and thus compact. Note that By the extreme value theorem, the infimum can be achieved by some .
Next, we show that the minimizer is unique. Suppose there are two points such that . Let . Since is convex, and thus . However, we have which yields that Thus the minimizer is unique, and is well-defined.

Lemma

Let be a nonempty, closed and convex set. Given and , for any , it holds that .

Proof

Note that for all , . So Thus, for all , which concludes that .

Corollary

Let be a nonempty, closed and convex set. For any , there exists such that

Geometrically, this means and can be strictly separated by a hyperplane. This is a special case of the separating hyperplane theorem we will discuss shortly.

Proof

Let , and . Since , . Then we have for any , which is equivalent to Taking the supremum over , it gives that

In fact, the hyperplane orthogonal to separates and . We can also generalize this lemma to two convex sets.

Theorem (Strictly separating hyperplane theorem)

Let be two disjoint closed convex sets, and at least one of them is bounded. Then there exists such that Namely, there exists and such that

The idea is to find such that , and show that (the hyperplane orthogonal to ) is a desired one.

4.2 Supporting hyperplane theorem

Definition

The interior of a set is defined as:
The closure of a set is defined as
The boundary of a set is defined as or equivalently,

Theorem (Supporting hyperplane theorem)

Given a nonempty convex set , and a point , there exists such that is a supporting hyperplane of at , namely,

Proof

If , then lies in an affine set of dimension less than . Otherwise, there exists affinely independent points in , which implies that contains a -simplex. However, the interior of the simplex is nonempty, which contradicts . Now choose any hyperplane that the affine set lies on and we are done.
If , let . Note that and is closed. By the corollary in Section 4.1, for all , there exists a hyperplane strictly separates and , namely, such that , . We normalize such that .
Next we consider a series of points For each , corresponds to a , and . Hence, by the Bolzano–Weierstrass theorem, there exists a convergent subsequence of . Denote by its limit. Then we show that this is the coefficient of the desired hyperplane.
For any , there exists such that Thus, by taking the limit on both sides.
For any , there exists a sequence by convexity. (Why?) Since for each , we can conclude that .

Proposition

Let be a convex set with nonempty interior, be a boundary point. Then there exists a sequence such that .

Proof

By definition, there exists such that . Since , choose any point , thus there exists such that . That is, for any with , . So by convexity, for any and , which implies that . Thus, .
Let . We have for all , and .

Corollary of the supporting hyperplane theorem

Any nonempty closed convex is the intersection of some halfspaces.

In fact, for any closed convex , We will not give the formal proof of this proposition in our lecture. However, let us try to understand this proposition intuitively.
For a -dimension convex set. We can find a tangent line at each boundary point, and the set only lies in a single side of the line. For all of these boundary points, we can get a lot of tangent lines, and an area bounded by these lines. Hence, the proposition tells us this area is just the original convex set.

When considering high dimensional spaces, we can just use the supporting hyperplane theorem. For each boundary point , let and make lie in the halfspace of . Thus, this proposition tells us that

Remark

Note that the number of those halfspaces may be infinite and even uncountable.

4.3 Separating hyperplane theorem

We would like to show that any two disjoint (not necessarily bounded or closed!) convex sets can be separated by a hyperplane. Note that the hyperplane may not separate these two sets strictly.

Theorem (Separating hyperplane theorem)

Let and be two disjoint convex sets. Then there exists a hyperplane separating and , namely, for all , and for all , .

Proof

Consider the set It suffices to separates and . This is because, if there exists a hyperplane such that , . Then for all and , we have . Finally, let .

Case 1: . By the corollary in Section 4.1, there is a hyperplane separating and .
Case 2: . Applying the supporting hyperplane theorem, we can find a supporting hyperplane for at , which separates and .

4.4 Farkas’ lemma

We now present an application of the separating hyperplane theorem. This lemma will help us prove the strong duality in Lecture 8.

Theorem (Farkas' lemma)

Let . Then exactly one of the following sets is empty:

;
.

Recall the conic combination and the cone hull.

The two sets can also be understood in the following ways:

The first set is non-empty means that locates in a cone hull of : Let . If there exists such that , then .
The second set is non-empty means that there exists such that the hyperplane separates and the column vectors of .
The Farkas’ lemma tells us there exists a separating hyperplane passing through unless .

Proof

First, we prove that if the first set is nonempty, the second one must be empty. Otherwise, there exist and such that: Next, we prove that if the first set is empty, the second one must be nonempty. It is easy to find a hyperplane to separate and by the strictly separting hyperplane theorem in Section 4.1. (Why?) Hence, there exists and such that The key problem is how to make the separating hyperplane pass through the original point. Actually, we can show that the hyperplane is also a separating hyperplane.

For all and , . Then , which is equivalent to . Taking the limit as , it gives .
In addition, implies that . Thus .

Therefore, the hypeplane is a desired hyperplane, and the second set is nonempty in this case.
Overall, exactly one of the two sets must be nonempty whenever the first one is empty or nonempty.

Question

Why is closed?