Ordinary Least Squares (OLS) — a step-by-step derivation

Beginner
Abstract

Where do the OLS formulas come from? A full derivation of the least-squares estimator in plain words: from intuition through diagrams to a step-by-step proof

The problem and the intuition

Consider a cloud of points — say years of experience (horizontal axis) and wage (vertical axis). The task is to draw a single straight line that “fits best”.

What does “best” mean? Every line is inexact somewhere — it passes near the points, not through them. The error for a single point is the vertical distance: the amount by which the line departs from the actual value.

wage yexperience xŷ = b₀ + b₁xeᵢ
Each red segment is an error (residual) — the vertical distance from a point to the line. OLS looks for the line whose sum of squared segments is the smallest.

The least-squares principle.

A line is “best” when its total departure from the data is smallest. Each error is squared (so that positive and negative values do not cancel, and large errors are penalised more heavily), the squares are summed, and the line minimising that sum is selected. Hence the name: the method of least squares.

Step 0: Notation

We have $n$ pairs of observations $(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)$. We fit a line:

$$ \hat{y}_i = b_0 + b_1 x_i $$

where $b_0$ is the intercept (where the line crosses the $y$-axis) and $b_1$ is the slope (how much $y$ rises when $x$ increases by 1).

The residual (error) for point $i$ is the difference between what we observe and what the line predicts:

$$ e_i = y_i - \hat{y}_i = y_i - b_0 - b_1 x_i $$
yᵢ (actual)ŷᵢ (predicted)eᵢxᵢyᵢŷᵢ
For one point: $y_i$ is the actual value, $\hat{y}_i$ is the value on the line, and the residual $e_i = y_i - \hat{y}_i$ is their difference.

Step 1: The objective function

The sum of squared residuals — denote it $S$:

$$ S(b_0, b_1) = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} \left(y_i - b_0 - b_1 x_i\right)^2 $$

This is a function of two unknowns, $b_0$ and $b_1$. We seek the values that minimise $S$.

The reason for squaring.

Summing the residuals $e_i$ themselves would let the positive and negative ones cancel — a poor measure. The absolute values $|e_i|$ are an alternative, but they are non-differentiable at zero and inconvenient analytically. The square is smooth, always positive, and penalises large errors more heavily. It also yields simple, closed-form formulas, derived below.

The function $S$ is a paraboloid — a bowl opening upwards. It has exactly one minimum, at the lowest point of the bowl, where the tangent is flat, i.e. the derivatives equal zero.

S(b₁)b₁minimum: dS/db₁ = 0b̂₁
The sum of squared residuals $S$ as a function of the slope $b_1$ is a parabola (a bowl). The minimum is where the tangent is flat — where the derivative = 0.

Step 2: The proof — first-order conditions

We find the minimum by taking the partial derivatives of $S$ with respect to $b_0$ and $b_1$ and setting them to zero.

Proof
Deriving the OLS formulas
  1. Derivative with respect to $b_0$. Differentiate $S$ in $b_0$ (chain rule — derivative of the square times the derivative of the inside, which is $-1$): $$ \frac{\partial S}{\partial b_0} = \sum_{i=1}^n 2\left(y_i - b_0 - b_1 x_i\right)(-1) = -2\sum_{i=1}^n \left(y_i - b_0 - b_1 x_i\right) $$
  2. Derivative with respect to $b_1$. The same, but the derivative of the inside in $b_1$ is $-x_i$: $$ \frac{\partial S}{\partial b_1} = -2\sum_{i=1}^n x_i\left(y_i - b_0 - b_1 x_i\right) $$
  3. Set both to zero. Divide by $-2$ and obtain the normal equations: $$ \sum_{i=1}^n \left(y_i - b_0 - b_1 x_i\right) = 0 \qquad\text{and}\qquad \sum_{i=1}^n x_i\left(y_i - b_0 - b_1 x_i\right) = 0 $$
  4. Solve the first for $b_0$. Split the sum and divide by $n$ (recalling $\frac{1}{n}\sum y_i = \bar{y}$ and $\frac{1}{n}\sum x_i = \bar{x}$): $$ \sum y_i = n b_0 + b_1 \sum x_i \;\;\Longrightarrow\;\; \boxed{\,b_0 = \bar{y} - b_1 \bar{x}\,} $$ This says something important: the OLS line always passes through the point of means $(\bar{x}, \bar{y})$.
  5. Substitute $b_0$ into the second equation. Plugging in $b_0 = \bar{y} - b_1\bar{x}$ and rearranging: $$ \sum x_i(y_i - \bar{y}) = b_1 \sum x_i(x_i - \bar{x}) $$
  6. Solve for $b_1$. Using the identity $\sum x_i(y_i-\bar y)=\sum (x_i-\bar x)(y_i-\bar y)$ (because $\sum \bar x (y_i - \bar y)=0$), we obtain the final formula: $$ \boxed{\,b_1 = \dfrac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\,} $$
∎
Theorem
The OLS estimator

The slope is the covariance divided by the variance of $x$, and the intercept pins the line to the point of means:

$$ b_1 = \frac{\mathrm{Cov}(x,y)}{\mathrm{Var}(x)} = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2}, \qquad b_0 = \bar{y} - b_1\bar{x} $$

Step 3: Verification that it is a minimum

A zero derivative marks a critical point — but is it a minimum or a maximum? The second derivative resolves this:

$$ \frac{\partial^2 S}{\partial b_1^2} = 2\sum_{i=1}^n x_i^2 > 0 $$

The second derivative is positive (a sum of squares), so the function is convex, and the critical point is indeed a minimum — a bowl opening upwards, as in the figure above.

Step 4: A worked numerical example

Five observations: experience $x$ (years) and wage $y$ (thousands).

$x_i$$y_i$$x_i-\bar{x}$$y_i-\bar{y}$$(x_i-\bar{x})(y_i-\bar{y})$$(x_i-\bar{x})^2$
13−2−244
24−1−111
350000
461111
572244
Σ1010

Means: $\bar{x} = 3$, $\bar{y} = 5$. Plugging into the formulas:

$$ b_1 = \frac{10}{10} = 1, \qquad b_0 = 5 - 1\cdot 3 = 2 $$

Thus $\hat{y} = 2 + 1\cdot x$ — each year of experience adds approximately 1 thousand on average. This may be verified in R / Python:

x <- c(1,2,3,4,5); y <- c(3,4,5,6,7)
coef(lm(y ~ x))     # (Intercept) 2,  x 1

import numpy as np
x = np.array([1,2,3,4,5]); y = np.array([3,4,5,6,7])
b1 = np.cov(x, y, bias=True)[0,1] / np.var(x)   # 1.0
b0 = y.mean() - b1 * x.mean()                    # 2.0

Step 5: The geometric view

There is a more elegant way to view OLS. Collecting all the $y_i$ into a single vector $\mathbf{y}$ in $n$-dimensional space, all possible fits $\hat{\mathbf{y}}$ lie on a plane (the space spanned by the columns of $\mathbf{X}$). OLS selects the point on that plane closest to $\mathbf{y}$ — that is, the orthogonal projection.

column space of Xyŷe
The geometry of OLS: $\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto the space of fits. The residual $\mathbf{e}$ is perpendicular to that plane — which is exactly why $\mathbf{X}^\top\mathbf{e}=0$.

The same normal equation $\sum x_i e_i = 0$ derived above with calculus states, geometrically: the residual is perpendicular to $x$ — the same result expressed in two languages.

Summary

Definition
Summary in one sentence
OLS picks the line by minimising the sum of squared vertical distances. The solution: $b_1 = \dfrac{\mathrm{Cov}(x,y)}{\mathrm{Var}(x)}$, $b_0 = \bar{y} - b_1\bar{x}$ — and the line always passes through $(\bar{x}, \bar{y})$.
  1. The residual is the vertical amount by which the line departs from the observation
  2. The sum of their squares is minimised (smooth, penalises large errors)
  3. Derivatives = 0 → normal equations → formulas
  4. Second derivative > 0 → it is certainly a minimum
  5. Geometrically → an orthogonal projection, residual ⟂ regressors

More in: Econometrics · Basics