As we have seen in the last lecture, a constrained optimization problem is equivalent to an unconstrained optimization problem with an indicator function. Moreover, since we have known that how to optimize a convex optimization problem with equality constraints (Newton’s method), we can only use the indicator function for inequality constraints. Given a convex optimization problem it is equivalent to where Unfortunately, the objective function is no longer smooth. To get a problem that we can actually solve, the idea is to approximate by so-called barrier functions.
Let be the interior of the feasible set for inequality constraints, namely, Assume is nonempty. Then a barrier function for is a continuous, strictly convex function such that as goes to the boundary of . Two common barrier functions are
logarithmic barrier function:
inverse barrier function: Note that both of them are convex. In this lecture, we will focus on the the logarithmic barrier function.
Example
Suppose the constraints are namely, the feasible set is . Then the barrier function can be
logarithmic barrier:
inverse barrier:
(, , the blue line is the log barrier and the orange line is the inverse barrier)
18.2 Central points and central path
The main idea is to approximate by the log barrier functions with coefficients and use the Newton method to solve the approximated optimization problem where the parameter controls the approximation accuracy.
Example
Suppose the constraints are namely, the feasible set is . Then the log barrier function is .
(, , the blue line is the case , the orange line is the case and the green line is the case )
Note even if . Thus, to guarantee the approximated problem is feasible, the original problem should be strictly feasible, namely, is nonempty.
We can solve the approximated problem by e.g. damped Newton’s method. We need always check whether during backtracking line search. This can be achieved either by modifying the condition of the while loop, or by defining for .
We will need the gradient and Hessian of log barrier functions. We state the results as a lemma.
Lemma
Let be the log barrier function. Then we have
The proof is a straight application of the chain rule. We leave it as an exercise.
The solution to the approximated problem using parameter is called a central point. As varies, the path traced out by is called the central path of the original optimization problem. Note is an interior point of the set defined by the inequality constraints, and it always satisfies . So it is always a strictly feasible solution to the original optimization problem. As , converges to the optimal solution of the original problem, as converges to .
18.3 KKT conditions and suboptimality gap of central points
Note that for the approximated problem using log barrier functions and parameter , if is the optimal solution, then Lagrange multipliers conditions assert that there exists such that that is, Let . Since , it gives the approximated KKT conditions for the original problem:
;
for all ;
;
for all and for all .
We now consider the suboptimality gap: . Since is feasible, and , we know that Fix and , and let . It is easy to see that is a convex function for . So the Lagrange conditions (approximated KKT conditions) of the approximated problem imply that In particular, it gives that Combining with we conclude the following suboptimality theorem.
Theorem
18.4 Barrier method
Since is -suboptimal, to achieve a desired accuracy , it suffices to solve the approximated problem with , namely, However, this works well only for small problems, good initial points, and moderate accuracy ( not too small). A better way is to use the barrier method, or path-following method, which chooses an increasing sequence and solves a sequence of problems, until , namely, When solving for the -th central point , it uses the central point as the initial point. This algorithm is called path-following since the iterates () “follow” the central path.
To guarantee the efficiency of our algorithm, we usually choose .
The final question is how to choose a strictly feasible . The idea is similar to the two-phase simplex algorithm. Consider the following relaxed problem Note that there exists a strictly feasible of the original problem iff the optimal value of the relaxed problem is negative. But how to solve this relaxed problem? We can use the barrier method now, since there exists a obvious strictly feasible solution where is a solution to the linear equations , which can be obtained by the Gaussian elimination, and . Moreover, it is not necessary to run the barrier method until optimality. Once we find a feasible with , we can stop and use as the initial point in the barrier method for the original problem.
