Learning path

Full curriculum

Full curriculum

Arrows go from each prerequisite to the units that depend on it. Hover or focus a unit to highlight its path.

Unit content

Capacity of a discrete memoryless channel

For a discrete memoryless channel, the mutual information $I(X;Y)$ depends on both the channel transition probabilities and the distribution used for the input symbols.

Its capacity is

$$C=\max_{p(x)}I(X;Y)$$

bits per channel use.

The maximization chooses how frequently each allowed input symbol should be used.

A completely uninformative channel has capacity zero because its output is independent of its input. A noiseless channel with $M$ distinguishable symbols has capacity

$$\log_2M$$

bits per use, achieved by using the symbols uniformly.

For a binary symmetric channel with crossover probability $p$,

$$C=1-H_2(p),$$

where

$$H_2(p)=-p\log_2p-(1-p)\log_2(1-p)$$

is binary entropy. Capacity falls from one bit per use at $p=0$ to zero at $p=1/2$.