Topic 7 of 8

Orthogonality and least squares

Step 19 · Linear algebra topics

Use dot products to find perpendicular directions and projections, build orthogonal bases with Gram–Schmidt, and fit the best line to data by least squares.

Orthogonal vectors and projections

Vectors are orthogonal when their dot product is 0. The projection of b onto a is the multiple of a closest to b; what is left over, b minus the projection, is orthogonal to a.

\text{proj}_{\mathbf{a}}\mathbf{b} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}\,\mathbf{a}

Example: A projection

Project b = ⟨3, 4⟩ onto a = ⟨1, 1⟩.

a · b = 7 and a · a = 2.
The projection is 3.5⟨1, 1⟩.
The leftover ⟨−0.5, 0.5⟩ has dot product 0 with ⟨1, 1⟩.

Answer⟨3.5, 3.5⟩

The projection of ⟨3, 4⟩ onto the line through ⟨1, 1⟩ is ⟨3.5, 3.5⟩. The dashed leftover meets the line at a right angle.

Orthogonal bases

In an orthogonal basis every pair of basis vectors is orthogonal. Then coordinates come from dot products alone, with no system to solve. Gram–Schmidt turns any basis into an orthogonal one by subtracting projections.

Example: Coordinates for free

u₁ = ⟨1, 1⟩ and u₂ = ⟨1, −1⟩ are orthogonal. Write ⟨5, 1⟩ in this basis.

The coefficient of u₁ is (⟨5, 1⟩ · u₁) ÷ (u₁ · u₁), which is 6 ÷ 2 = 3.
The coefficient of u₂ is 4 ÷ 2 = 2.

Answer⟨5, 1⟩ = 3u₁ + 2u₂

Example: Gram–Schmidt

Make v₁ = ⟨1, 1, 0⟩ and v₂ = ⟨1, 0, 1⟩ orthogonal.

Keep u₁ = v₁.
Take away from v₂ its projection onto u₁, which is one half of ⟨1, 1, 0⟩.
That leaves u₂ = ⟨0.5, −0.5, 1⟩.

Answer⟨1, 1, 0⟩ and ⟨0.5, −0.5, 1⟩

Least squares

When Ax = b has no solution, look for the x̂ that makes Ax̂ as close to b as possible. Then b − Ax̂ is orthogonal to the column space of A, which gives the normal equations.

A^{T}A\,\hat{\mathbf{x}} = A^{T}\mathbf{b}

Example: A best line

Fit y = c + mx to the points (0, 1), (1, 2) and (2, 4).

A has rows (1, 0), (1, 1) and (1, 2), and b = ⟨1, 2, 4⟩.
AᵀA = $\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}$ and Aᵀb = ⟨7, 10⟩.
Solve 3c + 3m = 7 and 3c + 5m = 10: m = 1.5 and c = 5/6.

Answery ≈ 0.83 + 1.5x

Try it yourself

Work each one on paper first, then open the answer to check.

Are ⟨2, 3⟩ and ⟨3, −2⟩ orthogonal?
Show answer
Yes: their dot product is 0
Project ⟨4, 2⟩ onto ⟨1, 0⟩.
Show answer
⟨4, 0⟩
Project ⟨2, 4⟩ onto ⟨1, 1⟩.
Show answer
⟨3, 3⟩
u₁ = ⟨1, 1⟩ and u₂ = ⟨1, −1⟩. What is the coefficient of u₁ for ⟨3, 7⟩?
Show answer
5
Write the normal equations for Ax = b.
Show answer
AᵀAx̂ = Aᵀb
Q has orthonormal columns. What is QᵀQ?
Show answer
I, the identity