Least Squares as an Orthogonal Projection

5 questions0m 0s

Prerequisites

Vectors, dot products, matrix transposes, and derivatives of quadratics.

Learn the skill

Minimize the squared residual norm ||Ax - b||^2. Differentiating gives A^T(Ax - b) = 0: the residual is orthogonal to the column space. Full column rank gives a unique coefficient vector; a fitted vector can remain unique without unique coefficients.

Worked example

For b = (1, 5) and A = (1, 1)^T, the best constant is 3. The residual b - Ax = (-2, 2) sums to zero, so it is perpendicular to A.

Model and Assumptions

Fit one constant x to invented observations b = (1, 2, 6). A is a three-by-one column of ones. Give all observations equal weight and define the residual as r = b - Ax.

1. What fitted constant minimizes the squared error? Answer

2. What is the sum of the three residuals at that fit? Answer

3. What is the minimized sum of squared residuals? Answer

4. Which condition characterizes a least-squares fit? r must equal b Every residual must be zero A^T r = 0

5. If A has dependent columns, what can still be unique? The choice of basis The fitted vector Ax Every coefficient vector x

Further inquiry

Derive the normal equations for a general real matrix. Then replace A with two identical columns of ones: describe all minimizing coefficient pairs and prove that their fitted vectors agree.

Review criteria

Differentiate or complete the square explicitly.
Give the coefficient family and identify its null direction.
Distinguish uniqueness of the projection from uniqueness of coefficients.

Research notes (not saved or graded)