Univariate Optimisation

Univariate optimisation deals with finding the minimum or maximum of a function with a single variable ($D=1$). While simpler than multivariate optimisation, it forms the foundation for many advanced optimisation techniques.

Mathematical Formulation

For a univariate function $L(\theta )$ where $\theta \in \mathbb{R}$, we want to find:

• ${\theta }^{\ast }=\arg \min L(\theta )$ for minimization, or
• ${\theta }^{\ast }=\arg \max L(\theta )$ for maximization

Optimality Conditions

Necessary Condition

For a smooth function, a necessary condition for an extremum (minimum or maximum) is:

$L^{\prime }({\theta }^{\ast })=0$

This means the derivative of the function at the optimal point equals zero.

Sufficient Conditions

To determine whether a stationary point (where $L^{\prime }(\theta )=0$) is a minimum, maximum, or neither:

Let’s say $L^{\prime }({\theta }^{\ast })=L^{\prime \prime }({\theta }^{\ast })=...=L^{(n-1)}({\theta }^{\ast })=0$, but $L^{(n)}({\theta }^{\ast })\ne 0$. Then:

• If $n$ is even and $L^{(n)}({\theta }^{\ast })>0$, then ${\theta }^{\ast }$ is a minimum
• If $n$ is even and $L^{(n)}({\theta }^{\ast })<0$, then ${\theta }^{\ast }$ is a maximum
• If $n$ is odd, then ${\theta }^{\ast }$ is neither a minimum nor a maximum (inflection point)

For most practical cases, checking the second derivative is sufficient:

• If $L^{\prime }({\theta }^{\ast })=0$ and $L^{\prime \prime }({\theta }^{\ast })>0$, then ${\theta }^{\ast }$ is a minimum
• If $L^{\prime }({\theta }^{\ast })=0$ and $L^{\prime \prime }({\theta }^{\ast })<0$, then ${\theta }^{\ast }$ is a maximum
• If $L^{\prime }({\theta }^{\ast })=0$ and $L^{\prime \prime }({\theta }^{\ast })=0$, higher-order derivatives must be checked

Analytical Methods

In some cases, we can solve for the optimum analytically:

1. Find all points where L’(θ) = 0
2. Check the second derivative to determine whether each point is a minimum or maximum
3. Compare function values at these points and at the boundaries to find the global optimum

Example: Projectile Motion

Consider the problem of finding the angle that maximizes the range of a projectile:

• Range function: $d=\frac{v_0^2}{g}\sin (2\theta )+\frac{v_0\cos (\theta )}{g}\sqrt{v_0^2{\sin }^2(\theta )+2Hg}$
• Analytical solution requires solving $d^{\prime }(\theta )=0$
• When $H=0$ (flat ground), the optimal angle is ${\theta }^{\ast }=45°$

Numerical Methods

Many univariate optimisation problems can’t be solved analytically. Numerical methods include:

Bracketing Methods

These methods narrow down the interval containing the optimum:

• Bisection Method: Repeatedly halves the interval
• Golden Section Search: Uses the golden ratio to reduce the interval
• Fibonacci Search: Uses Fibonacci numbers for interval reduction

Point-Based Methods

These methods generate a sequence of points converging to the optimum:

• Newton-Raphson Method: ${\theta }_{n+1}={\theta }_n-\frac{L^{\prime }({\theta }_n)}{L^{\prime \prime }({\theta }_n)}$
  • Requires first and second derivatives
  • Quadratic convergence near the solution
• Secant Method: Approximates the second derivative using finite differences
• Fixed-Point Iteration: ${\theta }_{n+1}=g({\theta }_n)$ where $g$ is constructed to converge to the optimum

Special Cases

Linear Functions

For $L(\theta )=a\theta +b$:

• If $a>0$, minimum at lower bound of domain
• If $a<0$, minimum at upper bound of domain
• If $a=0$, function is constant (no unique optimum)

Quadratic Functions

For $L(\theta )=a{\theta }^2+b\theta +c$:

• If $a>0$, unique minimum at ${\theta }^{\ast }=-b/(2a)$
• If $a<0$, unique maximum at ${\theta }^{\ast }=-b/(2a)$
• If $a=0$, reduces to linear case

Applications

Univariate optimisation appears in many engineering problems:

• Finding optimal timing in control systems
• Determining optimal thickness of a material
• Setting optimal temperature for a chemical reaction
• Finding optimal angle or position in mechanical systems

Related Pages

• Introduction to Optimisation
• Newton-Raphson Method
• Multivariate Optimisation
• Problem Formulation