Several random quantities can be collected into a random vector

$$\mathbf X=(X_1,\ldots,X_n)^T.$$

Its mean is the vector

$$\boldsymbol\mu=E[\mathbf X],$$

whose components are the expected values of the individual random variables.

Covariance matrix

The covariance matrix is

$$\Sigma=E[(\mathbf X-\boldsymbol\mu)(\mathbf X-\boldsymbol\mu)^T].$$

Its diagonal entries are variances, while off-diagonal entries are covariances:

$$\Sigma_{ij}=\operatorname{Cov}(X_i,X_j).$$

Geometry of uncertainty

For any vector $\mathbf a$,

$$\operatorname{Var}(\mathbf a^T\mathbf X)=\mathbf a^T\Sigma\mathbf a\ge0,$$

so covariance matrices are positive semidefinite.

The matrix therefore describes not only the uncertainty of each component but also which combinations of components tend to vary together.

Linear transformations

If

$$\mathbf Y=A\mathbf X+\mathbf b,$$

then

$$E[\mathbf Y]=A\boldsymbol\mu+\mathbf b,$$

and

$$\operatorname{Cov}(\mathbf Y)=A\Sigma A^T.$$

This transformation rule makes covariance matrices central to linear estimation and state-space models.