myLearning

Step 22: Probability theory

Law of large numbers and central limit theorem

Step 22 · Probability theory topics

See why averages of many independent trials settle near the mean, why their fluctuations follow a normal curve, and use that to approximate probabilities for sums.

The law of large numbers

Let X̄ₙ be the average of n independent copies of X, which has mean μ. The law of large numbers says X̄ₙ gets close to μ as n grows: the chance it misses by more than any fixed amount goes to 0.

P(∣Xˉn−μ∣>ε)→0 as n→∞P\left(|\bar{X}_n - \mu| > \varepsilon\right) \to 0 \text{ as } n \to \infty

Example: Chebyshev's inequality

Var(X̄ₙ) = σ²/n. With σ² = 4 and n = 400, bound P(|X̄ − μ| > 0.5).

  1. Chebyshev: P(∣Xˉ−μ∣>ε)≤Var(Xˉ)ε2P(|\bar{X} - \mu| > \varepsilon) \le \frac{\text{Var}(\bar{X})}{\varepsilon^{2}}.
  2. Var(X̄) = 4 ÷ 400 = 0.01, and 0.01 ÷ 0.25 = 0.04.

AnswerAt most 0.04

Example: Coin tosses

After many heads in a row, does the law say tails become more likely?

  1. No: every toss is still 50-50.
  2. The average settles because a huge number of later tosses swamps any early run, not because the coin corrects itself.

AnswerNo: there is no catching up

The central limit theorem

Take n independent copies of X, with mean μ and SD σ. Their sum Sₙ is approximately normal with mean nμ and SD σ√n, whatever the shape of X's own distribution. That is the central limit theorem.

Sn−nμσn≈N(0,1)\frac{S_n - n\mu}{\sigma\sqrt{n}} \approx N(0, 1)
Heads in 9 tosses of a coin: out of 512 equally likely sequences, the counts already form a bell shape.

Example: Dice

Roll 100 dice. What is the probability that the total is more than 370?

  1. One die has mean 3.5 and variance 35/12, about 2.92.
  2. The total has mean 350 and SD √292, about 17.1.
  3. z = (370 − 350) ÷ 17.1, about 1.17, and P(Z > 1.17) is about 0.12.

AnswerAbout 0.12

Normal approximations

A binomial count with n trials is a sum of n coin flips, so for large n it is nearly normal, with mean np and SD √(np(1 − p)). A common rule: use this when np and n(1 − p) are both at least 10. Moving the boundary by a half, the continuity correction, improves the fit.

Example: Coin tosses

Toss a fair coin 400 times. Approximate P(at least 220 heads).

  1. Mean 200 and SD √100, which is 10.
  2. With the continuity correction, use 219.5: z = 1.95.
  3. P(Z ≥ 1.95) is about 0.026.

AnswerAbout 0.026

Example: Polling

30% of a population favor a plan. In a random sample of 1,000, what is the chance at most 280 favor it?

  1. Mean 300 and SD √210, about 14.5.
  2. z = (280.5 − 300) ÷ 14.5, about −1.35.
  3. P(Z ≤ −1.35) is about 0.09.

AnswerAbout 0.09

Try it yourself

Work each one on paper first, then open the answer to check.

  1. The average of 10,000 fair die rolls is most likely close to what?

    Show answer

    3.5

  2. X has variance 9. What is the variance of the average of 100 independent copies?

    Show answer

    0.09

  3. X has mean 2 and SD 3. What are the mean and SD of the sum of 25 independent copies?

    Show answer

    Mean 50, SD 15

  4. Is the normal approximation good for a binomial with n = 20 and p = 0.01?

    Show answer

    No: np = 0.2 is far below 10

  5. Toss 100 fair coins. What are the mean and SD of the number of heads?

    Show answer

    Mean 50, SD 5

  6. Using that, about what is P(more than 60 heads)?

    Show answer

    About 0.02

What next?