1. Introduction to Optimization

Optimization is the science of making the best possible decision from a set of available options. It arises naturally in everyday life:

• A company wants to maximize profit while minimizing costs.

• A commuter wants to choose the fastest route under traffic conditions.

• A researcher wants to allocate limited computational resources to various tasks.

At its core, optimization is about trade-offs: between objectives, between feasibility and optimality, and between computational cost and solution quality.

Mathematically, an optimization problem seeks to minimize (or maximize) an objective function over a set of variables, often under constraints:

(1.1)\[\begin{split}\begin{aligned} \min_{\vw \in \mathcal{F}} \quad & f(\vw) \\ \text{s.t.} \quad & \vw\in \mathcal{F} \end{aligned}\end{split}\]

Here, \(\vw\in\mathbb{R}^d\) is the optimization variable, \(f(\cdot)\) is the objective function, \(\mathcal{F}\subseteq \mathbb{R}^d\) is the feasible set of the problem.

This general framework encompasses a vast array of real-world problems in machine learning, signal processing, control, and beyond.