A vector embedding represents a discrete item—such as a word, token, category or object—as a vector in a continuous space.

Instead of treating two symbols as unrelated labels, an embedding lets a model learn geometric relationships between their representations.

Embedding tables

For a finite vocabulary, the simplest learned embedding is a matrix whose rows are vectors. Looking up item $i$ selects one row:

$$\mathbf e_i=E[i].$$

During training, the entries of $E$ are adjusted like other model parameters.

Geometry

Distances, dot products and directions in embedding space can encode useful statistical relationships learned from data.

The coordinates themselves usually have no fixed human-readable meaning. What matters is how vectors relate to one another and how later layers use them.

Tokens and positions

Sequence models often combine a token embedding with information about where that token occurs. Without some positional representation, a self-attention layer has no inherent notion of first, second or later positions.

Embeddings are learned representations

An embedding does not guarantee semantic similarity merely because vectors are close. Its geometry reflects the training objective and data that produced it.

Embeddings provide the continuous vector inputs on which neural sequence models can apply linear algebra and attention.