Learning path

Full curriculum

Full curriculum

Arrows go from each prerequisite to the units that depend on it. Hover or focus a unit to highlight its path.

Unit content

Step sizes and line search in descent methods

A descent direction says where to move; a step size says how far.

Given a current point $x_k$ and descent direction $p_k$, an update has the form

$$x_{k+1}=x_k+\alpha_k p_k,$$

where $\alpha_k>0$ is the step size.

A step that is too small wastes iterations. A step that is too large can cross a useful valley, increase the objective or make the iteration diverge.

A line search chooses $\alpha_k$ by examining the one-dimensional function

$$\phi(\alpha)=f(x_k+\alpha p_k).$$

Practical line searches usually seek sufficient decrease rather than the exact minimum along the line. Backtracking, for example, repeatedly shrinks a trial step until it reduces the objective enough.

Step-size selection is therefore part of the optimization algorithm, not a cosmetic tuning detail.