Multivariate Optimisation

Multivariate optimisation involves finding the minimum or maximum of a function with multiple variables ($D\ge 2$). Most real-world engineering problems involve multiple design variables, making multivariate optimisation essential.

Mathematical Formulation

For a function $L(\theta )$ where $\theta \in {\mathbb{R}}^D$, we seek:

• ${\theta }^{\ast }=\arg \min L(\theta )$ for minimization, or
• ${\theta }^{\ast }=\arg \max L(\theta )$ for maximization

Key Concepts

Gradient Vector

The gradient vector ($\mathbf{g}$) is the vector of first partial derivatives:

$\mathbf{g}={\nabla }_{\theta }L={\left[\frac{\partial L}{\partial {\theta }_1},\frac{\partial L}{\partial {\theta }_2},\dots ,\frac{\partial L}{\partial {\theta }_D}\right]}^T$

The gradient points in the direction of steepest increase of the function.

Example:

$$\mathcal{L}=\delta =\frac{64F{L}^3}{3\pi E{D}^4}$$ $$\underset{\text{vector}}{\underbrace{\tilde{\delta }}}={\nabla }_{(D,L)}\mathcal{L}=\left[\begin{array}{c}\frac{\partial \mathcal{L}}{\partial D}\\ \frac{\partial \mathcal{L}}{\partial L}\end{array}\right]=\left[\begin{array}{c}-\frac{256F{L}^3}{3\pi E{D}^5}\\ \frac{192F{L}^2}{3\pi E{D}^4}\end{array}\right]$$

Hessian Matrix

The Hessian matrix ($\mathbf{H}$) is the matrix of second partial derivatives:

$\mathbf{H}={\nabla }_{\theta }^{2}L\text{ where }[\mathbf{H}{]}_{ij}=\frac{{\partial }^{2}L}{\partial {\theta }_i\partial {\theta }_j}$

The Hessian characterizes the local curvature of the function.

Example:

$$\underset{\text{matrix}}{\underbrace{\tilde{H}}}={\nabla }_{(D,L)}^2\mathcal{L}=\left[\begin{array}{cc}\frac{{\partial }^2\mathcal{L}}{\partial D^2}& \frac{{\partial }^2\mathcal{L}}{\partial D\partial L}\\ \frac{{\partial }^2\mathcal{L}}{\partial L\partial D}& \frac{{\partial }^2\mathcal{L}}{\partial L^2}\end{array}\right]=\left[\begin{array}{cc}\frac{1280F{L}^3}{3\pi E{D}^6}& -\frac{256F{L}^2}{\pi E{D}^5}\\ -\frac{256F{L}^2}{\pi E{D}^5}& \frac{128FL}{\pi E{D}^4}\end{array}\right]$$

Note that this is a symmetric matrix.

Optimality Conditions

Necessary Conditions

For a smooth function, a necessary condition for a local minimum or maximum is:

${\nabla }_{\theta }L({\theta }^{\ast })=0$

This means the gradient equals the zero vector at the optimal point.

Sufficient Conditions

To determine the nature of a stationary point (where ${\nabla }_{\theta }L(\theta )=0$):

• If the Hessian $\mathbf{H}$ is positive definite (all eigenvalues positive), then ${\theta }^{\ast }$ is a local minimum
• If the Hessian $\mathbf{H}$ is negative definite (all eigenvalues negative), then ${\theta }^{\ast }$ is a local maximum
• If the Hessian $\mathbf{H}$ has both positive and negative eigenvalues, then ${\theta }^{\ast }$ is a saddle point
• If the Hessian $\mathbf{H}$ is singular (has zero eigenvalues), higher-order tests are needed

Special Cases

Linear Functions

For $L(\theta )={\mathbf{c}}^T\theta +b$:

• No extrema exist unless constrained
• Gradient $\nabla L=\mathbf{c}$ is constant
• Hessian is the zero matrix

Quadratic Functions

For $L(\theta )=\frac{1}{2}{\theta }^T\mathbf{Q}\theta +{\mathbf{c}}^T\theta +b$ where $\mathbf{Q}$ is symmetric:

• If $\mathbf{Q}$ is positive definite, unique minimum at ${\theta }^{\ast }=-{\mathbf{Q}}^{-1}\mathbf{c}$
• If $\mathbf{Q}$ is negative definite, unique maximum at ${\theta }^{\ast }=-{\mathbf{Q}}^{-1}\mathbf{c}$
• If $\mathbf{Q}$ is indefinite, saddle point at ${\theta }^{\ast }=-{\mathbf{Q}}^{-1}\mathbf{c}$
• If $\mathbf{Q}$ is singular, no extrema or infinitely many extrema

Numerical Methods

Gradient-Based Methods

• Steepest Descent:

  • Update: ${\theta }_{k+1}={\theta }_k-{\alpha }_k\nabla L({\theta }_k)$
  • Simple but can be slow, especially for ill-conditioned problems
  • Step size ${\alpha }_k$ determined by line search

• Conjugate Gradient:

  • Uses conjugate directions to improve convergence
  • Particularly effective for quadratic functions
  • Avoids the “zig-zag” behavior of steepest descent

Hessian-Based Methods

• Newton’s Method:

  • Update: ${\theta }_{k+1}={\theta }_k-{\mathbf{H}}^{-1}({\theta }_k)\nabla L({\theta }_k)$
  • Quadratic converg

• Quasi-Newton Methods:

  • Approximates the Hessian or its inverse
  • BFGS (Broyden-Fletcher-Goldfarb-Shanno) is the most popular
  • Avoids explicit computation of second derivatives

• Trust Region Methods:

  • Restricts the optimization step within a “trusted” region
  • Combines advantages of gradient and Hessian approaches
  • Robust for non-convex functions

Challenges in Multivariate Optimisation

Scaling

When different variables have vastly different scales, optimization algorithms may perform poorly. Solutions include:

• Variable transformation
• Preconditioning
• Normalized coordinates

Ill-Conditioning

When the Hessian has a high condition number (ratio of largest to smallest eigenvalue), convergence can be slow. This occurs in:

• Problems with highly interdependent variables
• Problems with vastly different sensitivities to different variables

Multiple Local Optima

Unlike in the convex case, general nonlinear functions may have multiple local optima, making it difficult to find the global optimum. Strategies include:

• Multiple random starts
• Global optimization methods
• Problem reformulation

Applications

Multivariate optimization appears in countless engineering applications:

• Structural design (multiple dimensional parameters)
• Control system tuning (multiple control parameters)
• Machine learning (model parameters)
• Financial portfolio optimization (multiple assets)
• Chemical process optimization (multiple process variables)

Related Pages

• Gradient-Based Methods
• Newton-Raphson Method
• Quasi-Newton Methods
• Trust Region Methods
• Scaling and Conditioning