Load balancing is something most of us assume is a solved problem. But the idea that load balancing is “solved” could not be further from the truth. If you use multiple load balancers, the problem is even worse. Most of us use “random” or “round-robin” techniques, which have certain advantages but are highly inefficient. Others use more complex algorithms like “least-conns,” which can be more efficient but have horrific edge cases. “Consistent hashing” is a very useful technique but only applies to certain problems.
There are several factors that exist both in theory and practice that make efficient load balancing an exceptionally hard problem, including Poisson request arrival times, exponentially distributed response latency, and oscillations when sharing data between multiple load balancers. Luckily, there are techniques and algorithms that have been developed that can make life better. Tyler McMullen explains some of the ways that we can do better than “random,” “round-robin,” and naive “least-conns,” even with distributed load balancers.
6. SLIDELOAD BALANCING IS IMPOSSIBLE 6
Abstraction Balancing LoadFailure
Treat many servers as one
Single entry point
Simplification
Transparent failover
Recover seamlessly
Simplification
Spread the load efficiently across servers
Why Load Balance?
Three major reasons. The least of which is balancing load.
7. R A N D O M
T H E I N G L O R I O U S D E FA U LT
A N D B A N E O F M Y E X I S T E N C E
8. SLIDE
LOAD BALANCING IS IMPOSSIBLE
What’s good about
random?
8
• Simplicity
• Few edge cases
• Easy failover
• Works identically when distributed
9. SLIDE
LOAD BALANCING IS IMPOSSIBLE
What’s bad about
random?
9
• Latency
• Especially long-tail latency
• Useable capacity
23. The probability of a single resource request avoiding the 99th percentile is 99%.
The probability of all N resource requests in a page avoiding
the 99th percentile is (99% ^ N).
99% ^ 69 = 49.9%
30. L E T ’ S T H R O W A W R E N C H I N T O T H I S . . .
D I S T R I B U T E D L O A D
B A L A N C I N G
A N D W H Y I T M A K E S E V E R Y T H I N G H A R D E R