Given a Sample mean , how close is to the real mean ?

It turns out that as increases, will increasingly get closer to , and the distribution of gets closer to Normal.

Weak law of large numbers (WLLN)

Given a sample mean with finite Expectation , for some , we have

The probability that gets close to goes to 100% when approaches infinity, no matter what definition of we choose, whether 0.00001 or 1000.

  • We say that converges in probability to .
  • WLLN holds for both finite and infinite variance (but is much harder to prove for the latter).

Proof of correctness

From Finding the variance, we know that , where for any . So .

Given the derivations above and a constant , Chebyshev’s inequality tells us

The is not part of the original inequality, but it must be true because probabilities are not negative.

We can now analyze this:

  • As , the denominator goes to infinity and the overall probability goes to zero. This is the probability that is greater than .
  • This means that approaches a probability of 100%.

Strong law of numbers (SLLN)

An even stronger claim can be made about WLLN. In particular, the following claim is true:

which completely eliminates the need for an and directly states that as the sample size approaches infinity, the sample mean will equal the population mean with 100% certainty.

  • We say that converges almost surely (with probability 1) to .