Round Robin
The simplest algorithm. Requests go to servers in order: A, B, C, A, B, C.
Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A
Works when all servers are identical and all requests cost roughly the same. Breaks down when one server is slower than the others or when some requests are expensive (file uploads, report generation).
Weighted Round Robin
Same as round robin, but servers with higher capacity get more requests:
upstream api {
server 10.0.1.1 weight=5; # 2x CPU
server 10.0.1.2 weight=3; # 1.5x CPU
server 10.0.1.3 weight=2; # baseline
}
Useful when your fleet is heterogeneous — some instances have more CPU or memory than others.
Least Connections
Routes each request to the server with the fewest active connections. This naturally adapts to slow servers: a server that is processing a slow request accumulates fewer new requests.
upstream api {
least_conn;
server 10.0.1.1;
server 10.0.1.2;
server 10.0.1.3;
}
This is my default choice for most API backends. It handles variable request costs better than round robin with no extra configuration.
IP Hash
Routes requests from the same client IP to the same server. Useful for sticky sessions when your app stores session state in memory:
upstream api {
ip_hash;
server 10.0.1.1;
server 10.0.1.2;
}
The downside: if one server goes down, all clients pinned to it get redistributed. If your sessions are in Redis or a database, you do not need IP hash — use least connections instead.
Random with Two Choices
Pick two servers at random, send the request to the one with fewer connections. This sounds crude but it is surprisingly effective. The "power of two choices" is a well-studied phenomenon — it avoids the herd behavior that plagues pure least-connections in large clusters.
Nginx does not support this natively, but Envoy and HAProxy do.
Health checks matter more than the algorithm
The best algorithm in the world is useless if it routes traffic to dead servers. Active health checks probe each server periodically:
upstream api {
least_conn;
server 10.0.1.1 max_fails=3 fail_timeout=30s;
server 10.0.1.2 max_fails=3 fail_timeout=30s;
server 10.0.1.3 max_fails=3 fail_timeout=30s;
}
After 3 failures in 30 seconds, Nginx marks the server as down and stops routing to it. After 30 seconds, it tries again.
Layer 4 vs Layer 7
- Layer 4 (TCP): routes based on IP and port. Faster, lower overhead. Cannot inspect HTTP headers or paths.
- Layer 7 (HTTP): routes based on URL paths, headers, cookies. More flexible. Slightly higher latency.
For most web apps, Layer 7 is the right choice because you need path-based routing (/api to backend, / to frontend). For high-throughput internal services, Layer 4 reduces overhead.
My recommendation
Start with least connections behind Nginx or an AWS ALB. It handles 95 percent of use cases. Only reach for something more complex when you have evidence that the simple approach is not working.