S3U

Math / Algebra / Graduate / mle01

Identifiability and Ridge Regularization

Analyze what observations identify before interpreting a stabilized estimate.

All worksheets
5 questions0m 0s

Prerequisites

Linear algebra, nullspaces, least squares, and quadratic optimization.

Learn the skill

A null direction changes parameters without changing predictions. A penalty can select one estimate but does not add observational information. Separate identification by data from selection by a regularization rule.

Worked example

If only x1 + x2 = 8 is observed, (3,5) and (5,3) fit equally well. The minimum-norm exact fit is (4,4), selected by symmetry and the norm rather than by separate measurements.

Model and Assumptions

An invented noiseless model observes y = x1 + x2 = 6. For lambda > 0, minimize J = (x1 + x2 - 6)^2 + lambda(x1^2 + x2^2). No nonnegativity constraint is imposed.

1. Which direction leaves the observation unchanged?
2. At lambda = 1, which pair minimizes J?
3. As lambda tends to zero from above, which pair is selected?
4. What does a unique penalized solution establish?
5. Which second exact observation identifies both coefficients?

Further inquiry

Derive the minimizer for arbitrary positive lambda. Compare its prediction bias with the unpenalized solution family. Design a second measurement and prove the combined matrix has full rank.

Review criteria

  • Show x1 = x2 = 6/(2 + lambda).
  • Explain why stabilization does not resolve data-only identifiability.
  • Check independence of the proposed measurement.