Parallel programs often need communication patterns involving an entire group of workers rather than one sender and one receiver. These are collective operations.

Common collectives include:

• broadcast: one worker sends the same value to all others;
• scatter: one worker distributes different pieces of a dataset;
• gather: pieces from all workers are collected at one worker;
• reduce: values from all workers are combined with an associative operation;
• all-reduce: the reduced result is delivered to every worker.

A naive broadcast from one root to $p-1$ workers can make the root a bottleneck. A tree broadcast instead lets recipients forward the value, reducing the number of sequential communication rounds to roughly $O(\log p)$.

Reductions can use the same tree structure in reverse. All-reduce can combine a reduction phase with redistribution of the result.

Collectives expose intent at the algorithm level. A parallel runtime or communication library can then choose an implementation suited to the machine topology and message sizes.

Their cost is not free: frequent global collectives can synchronize otherwise independent workers and become the scalability limit of a distributed computation.