Model Fitting and Regression

Model fitting is the process of identifying mathematical or computational models that best explain observed data. Regression, a common type of model fitting, focuses specifically on estimating relationships between variables. These techniques form the foundation of data-driven engineering and scientific discovery.

Fundamental Concepts

Model Components

A typical model fitting process involves:

1. Observed Data: A set of measurements or observations $(x_i,y_i)$ for $i=1,2,...,N$
2. Model Function: A mathematical relationship $y=f(x;\theta )$ parameterized by $\theta$
3. Error Measure: A quantification of the discrepancy between observations and model predictions
4. Optimization Problem: Finding the parameters $\theta$ that minimize the error measure

Types of Models

Models can be classified based on their structure:

1. Linear Models: $f(x;\theta )={\theta }_1{\varphi }_1(x)+{\theta }_2{\varphi }_2(x)+...+{\theta }_n{\varphi }_n(x)$

   • Linear in parameters $\theta$, not necessarily linear in $x$
   • Examples: linear regression, polynomial regression, basis function models

2. Nonlinear Models: $f(x;\theta )$ is nonlinear in parameters $\theta$

   • Examples: exponential models, logistic models, power laws, neural networks

3. Parametric vs. Nonparametric Models:

   • Parametric: Fixed form with finite number of parameters
   • Nonparametric: Flexible form, potentially infinite parameters (e.g., kernel methods)

Mathematical Formulation

General Problem Statement

Given data points $(x_i,y_i)$ for $i=1,2,...,N$, find the parameter vector $\theta$ that minimizes:

$E(\theta )={\sum }_{i=1}^{N}e(y_i,f(x_i;\theta ))$

where $e(y,\hat{y})$ is an error function measuring the discrepancy between observed $y$ and predicted $\hat{y}$.

Common Error Functions

1. Squared Error: $e(y,\hat{y})=(y-\hat{y}{)}^2$

   • Leads to least squares regression
   • Sensitive to outliers
   • Optimal for Gaussian noise

2. Absolute Error: $e(y,\hat{y})=|y-\hat{y}|$

   • Leads to least absolute deviations (LAD) regression
   • More robust to outliers
   • Optimal for Laplace distributed noise

3. Huber Loss: A hybrid approach

   • $e(y,\hat{y})=\frac{1}{2}(y-\hat{y}{)}^2$ for $|y-\hat{y}|\le \delta$
   • $e(y,\hat{y})=\delta |y-\hat{y}|-\frac{1}{2}{\delta }^2$ for $|y-\hat{y}|>\delta$
   • Combines robustness of absolute error with smoothness of squared error

Linear Regression

Simple Linear Regression

The simplest form of regression models the relationship between two variables with a straight line:

$y={\theta }_0+{\theta }_{1}x+ϵ$

where $ϵ$ represents random error.

Multiple Linear Regression

Extends simple linear regression to multiple independent variables:

$y={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_p{x}_p+ϵ$

Matrix Formulation

The multiple linear regression can be expressed in matrix notation:

$\mathbf{y}=\mathbf{X}\theta +ϵ$

where:

• $\mathbf{y}\in {\mathbb{R}}^N$ is the vector of observed dependent variables
• $\mathbf{X}\in {\mathbb{R}}^{N\times (p+1)}$ is the design matrix containing the independent variables
• $\theta \in {\mathbb{R}}^{p+1}$ is the parameter vector
• $ϵ\in {\mathbb{R}}^N$ is the error vector

Least Squares Solution

Under the least squares criterion, the optimal parameter vector is:

${\theta }^{\ast }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$

This closed-form solution exists when ${\mathbf{X}}^T\mathbf{X}$ is invertible.

Polynomial Regression

A specific case of linear regression where basis functions are powers of $x$:

$y={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+...+{\theta }_p{x}^p+ϵ$

This is linear in parameters $\theta$ despite modeling nonlinear relationships in $x$.

Nonlinear Regression

Formulation

Nonlinear regression involves models where the parameters appear nonlinearly:

$y=f(x;\theta )+ϵ$

Common examples include:

• Exponential models: $f(x;\theta )={\theta }_1{e}^{{\theta }_{2}x}$
• Growth models: $f(x;\theta )=\frac{{\theta }_{1}x}{{\theta }_2+x}$
• Sinusoidal models: $f(x;\theta )={\theta }_1\sin ({\theta }_{2}x+{\theta }_3)$

Optimization Approaches

Unlike linear regression, nonlinear regression typically requires iterative numerical optimization:

1. Gauss-Newton Method: Specialized for least squares problems
2. Levenberg-Marquardt Algorithm: Robust modification of Gauss-Newton
3. Gradient Descent: Simple but potentially slow convergence
4. Trust Region Methods: Restrict optimization steps to regions where the model is trusted

Regularization Techniques

Regularization addresses overfitting by adding penalties to the objective function:

Ridge Regression (L2 Regularization)

$E(\theta )={\sum }_{i=1}^N(y_i-f(x_i;\theta ){)}^2+\lambda {\sum }_{j=1}^p{\theta }_j^2$

• Shrinks parameters toward zero
• Handles multicollinearity well
• Closed-form solution: ${\theta }^{\ast }=({\mathbf{X}}^T\mathbf{X}+\lambda \mathbf{I}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$

Lasso Regression (L1 Regularization)

$E(\theta )={\sum }_{i=1}^N(y_i-f(x_i;\theta ){)}^2+\lambda {\sum }_{j=1}^p|{\theta }_j|$

• Produces sparse solutions (feature selection)
• No closed-form solution, requires quadratic programming
• Effective for high-dimensional data

Elastic Net

Combines L1 and L2 regularization:

$E(\theta )={\sum }_{i=1}^N(y_i-f(x_i;\theta ){)}^2+{\lambda }_1{\sum }_{j=1}^p|{\theta }_j|+{\lambda }_2{\sum }_{j=1}^p{\theta }_j^2$

Model Evaluation and Selection

Performance Metrics

1. Mean Squared Error (MSE): $\frac{1}{N}{\sum }_{i=1}^N(y_i-f(x_i;\theta ){)}^2$
2. Root Mean Squared Error (RMSE): $\sqrt{\text{MSE}}$
3. Mean Absolute Error (MAE): $\frac{1}{N}{\sum }_{i=1}^N|y_i-f(x_i;\theta )|$
4. Coefficient of Determination (R²): $1-\frac{{\sum }_i(y_i-f(x_i;\theta ){)}^2}{{\sum }_i(y_i-\bar{y}{)}^2}$
   • Represents the proportion of variance explained by the model
   • Ranges from 0 to 1 for linear models (can be negative for nonlinear models)
   • Higher values indicate better fit

Cross-Validation

Techniques to assess model performance on unseen data:

1. k-fold Cross-Validation: Divides data into k subsets, trains on k-1 subsets and tests on the remaining one, rotating through all subsets
2. Leave-One-Out Cross-Validation: Special case of k-fold where k equals the number of data points
3. Train-Test Split: Divides data into training and testing sets (typically 70-30% or 80-20%)
4. Hold-out Validation: Sets aside a portion of data for final evaluation

Model Selection Criteria

Metrics that balance goodness-of-fit with model complexity:

1. Akaike Information Criterion (AIC): $\text{AIC}=2k-2\ln (L)$
   • $k$ is the number of parameters
   • $L$ is the likelihood function value
2. Bayesian Information Criterion (BIC): $\text{BIC}=k\ln (n)-2\ln (L)$
   • Penalizes complexity more severely than AIC
3. Adjusted R²: ${\text{Adjusted R}}^2=1-\frac{(1-R^2)(N-1)}{N-p-1}$
   • Adjusts R² based on the number of predictors

Parameter Uncertainty and Confidence

Parameter Covariance Matrix

For least squares estimation, the parameter covariance matrix is:

$\text{Cov}(\theta )={\sigma }^2({\mathbf{X}}^T\mathbf{X}{)}^{-1}$

where ${\sigma }^2$ is the variance of the error term.

Confidence Intervals

For parameter ${\theta }_j$, the 95% confidence interval is:

${\theta }_j\pm t_{N-p-1,0.975}\cdot \text{SE}({\theta }_j)$

where $\text{SE}({\theta }_j)=\sqrt{\text{Cov}(\theta {)}_{jj}}$ and $t_{N-p-1,0.975}$ is the t-statistic with $N-p-1$ degrees of freedom.

Prediction Intervals

For a new observation at $x_0$, the prediction interval is:

$f(x_0;\theta )\pm t_{N-p-1,0.975}\cdot \sigma \sqrt{1+x_0^T({\mathbf{X}}^T\mathbf{X}{)}^{-1}{x}_0}$

Specialized Regression Techniques

Weighted Least Squares

Incorporates varying reliability of observations:

$E(\theta )={\sum }_{i=1}^N{w}_i(y_i-f(x_i;\theta ){)}^2$

where $w_i$ are weights, often inversely proportional to the variance of observations.

Robust Regression

Methods less sensitive to outliers:

1. M-Estimation: Minimizes a robust loss function
2. MM-Estimation: Combines high breakdown point with efficiency
3. Least Trimmed Squares: Minimizes the sum of the smallest residuals

Bayesian Regression

Incorporates prior knowledge about parameters:

$p(\theta |data)\propto p(data|\theta )\cdot p(\theta )$

• Prior: $p(\theta )$ represents knowledge before seeing data
• Likelihood: $p(data|\theta )$ represents how well parameters explain data
• Posterior: $p(\theta |data)$ represents updated knowledge after seeing data

Implementation with Optimization Algorithms

Model fitting is fundamentally an optimization problem. Common approaches include:

1. Direct Methods: For linear models with closed-form solutions
2. Iterative Methods: Required for nonlinear models
   • Gauss-Newton: Efficient for moderate nonlinearity
   • Levenberg-Marquardt: More robust, combines Gauss-Newton with steepest descent
   • Trust Region Methods: Controls step size based on model reliability
   • Stochastic Gradient Descent: Effective for large datasets

Applications in Engineering

1. System Identification: Determining mathematical models of dynamic systems
2. Empirical Modeling: Creating models based on experimental data
3. Parameter Estimation: Determining physical parameters from measurements
4. Response Surface Methodology: Optimizing processes using experimental designs
5. Calibration: Adjusting simulation models to match observed behavior

Related Pages

• Least Squares
• Gauss-Newton Method
• Levenberg-Marquardt Method
• Numerical Methods
• Linear Programming
• Constrained Optimization