Unit content
Load balancing in parallel computations
A parallel computation finishes when its slowest required worker finishes. If work is distributed unevenly, processors that finish early sit idle while one overloaded worker determines the total time.
For $p$ workers with assigned execution times $L_1,\ldots,L_p$, the parallel phase cannot finish before
$$T\ge \max_i L_i.$$
Equal task counts do not guarantee equal load. Ten image tiles can have very different rendering cost, and graph vertices can have very different numbers of neighbors.
Static scheduling assigns work in advance. It has low overhead and works well when task costs are predictable and uniform.
Dynamic scheduling assigns work as workers become available. It adapts to irregular costs but introduces queues, synchronization and less predictable locality.
Chunk size couples load balance to granularity. Large chunks reduce scheduling overhead but can leave a long tail; small chunks distribute irregular work more evenly but cost more to manage.
Good load balancing minimizes idle resources without turning scheduling itself into the bottleneck.