Lecture 1. What is Optimization

1.1 Introductory examples

What is optimization? Roughly speaking, optimization is to minimize or maximize a function (which is called the objective function) under some constraints.

For example, we some several ways to return the campus from Hongqiao Station: by taxi (Didi / Gaode), by metro or by bus (Hongqiao 4 Line / Min-Hong 2 Line), etc. We would like to minimize the time, but our money is limited. This is an optimization problem.

Formally, an optimization problem can be defined by where is called the objective function and is called the feasible set, usually specified by constraint functions The optimal solution is usually denoted by In this course, we consider continuous optimization problem, where the objective function and the constraints are continuous functions. We now give some more examples.

Knapsack problem


Suppose there are types of items. The -th type has volume , weight and value . We have a knapsack to bring some items. However the capacity of this knapsack is and the load-bearing is . That is, the total volume of the items in the knapsack can not exceed and the total weight of the items in the knapsack can not exceed . What is the maximum value we can bring?

For each , define a variable to denote the number of carried items of the -th type. Then we can formalize the problem as How can we solve this problem? For simplicity, we assume that there are only two types: Cola and potato chips. Each Cola has volume , weight and value ; each potato chips has volume , weight and value . Our knapsack has capacity and load-bearing . Now the problem is Actually, we can solve this problem by drawing a graph:
Pasted image 20230822011151.png

  1. What if we require integer ?
  2. What if there are more types?

Data fitting


Consider the free falling motion. The height and the time of a free fall follow the law . However, the practical data may not exhibit the perfect law.
Suppose we have the following data and we would like to use to fit the data. Which value of coefficients should we choose?

10 20 30 40
1.011 2.019 3.032 4.041

However, before solving this problem, we should first ask the following question: if we choose a certain value of , how can we measure the difference between the theoretical values of and the practical data?


Generally, we have the following question. Let , , where . Given a set of data and we guess the values of and , how can we measure the difference between and , where ?

If we only have two numbers and , it is natural to use the absolute value to measure the difference. Moreover, it is clear that is closer to than . However, if we have two vectors, how can we measure the difference? Is closer to than ?

We need to extend the concept of the absolute value to measure the distances between vectors in .

Definition (Norm)

Given a vector space over a field (usually ), a norm is a function having the following properties:
1. (Nonnegativity) , .
2. (Positive definiteness) iff .
3. (Absolute homogeneity) and , .
4. (Triangle inequality) </