The law of large numbers says that sample averages concentrate near the expected value. The central limit theorem describes the approximate shape of their remaining fluctuations.

Let $X_1,\ldots,X_n$ be independent identically distributed random variables with finite mean $\mu$ and variance $\sigma^2$. Their sample mean is

$$\bar X=\frac1n\sum_{i=1}^n X_i.$$

For large $n$, the standardized quantity

$$\frac{\bar X-\mu}{\sigma/\sqrt n}$$

is approximately standard normal under broad conditions.

Center and standard error

The sample mean has expected value

$$E[\bar X]=\mu$$

and standard deviation

$$\operatorname{SD}(\bar X)=\frac{\sigma}{\sqrt n}.$$

This standard deviation is the standard error of the sample mean.

Increasing the sample size by a factor of four therefore halves the standard error.

Why the theorem matters

The individual observations do not need to follow a normal distribution. Averages of many independent observations can nevertheless have an approximately normal sampling distribution.

This is why normal approximations appear throughout statistical inference.

Central limit theorem versus law of large numbers

The law of large numbers says where the sample mean goes:

$$\bar X\to\mu.$$

The central limit theorem describes the scale and approximate shape of the fluctuations around $\mu$:

$$\bar X-\mu\text{ is typically of order }\frac1{\sqrt n}.$$

The approximation can be poor for small samples from strongly skewed or heavy-tailed distributions, so the theorem's conditions and sample size still matter.