Load Balancing: The Traffic Controller of Modern Infrastructure

Load balancing exists because every system has a breaking point. Push enough traffic toward a single server and it buckles. The load balancer sits between users and your server pool, spreading requests around so nothing collapses.

I think of load balancers as air traffic control for your network. They do not just route packets. They make decisions based on real-time conditions, health status, and configured policies. Without them, scaling beyond a handful of servers becomes a nightmare of manual failover and prayer.

Core Concepts

A load balancer accepts incoming traffic and picks which backend server handles each request. The client only sees one destination IP address. The load balancer keeps up appearances while doing the actual work behind the scenes.

The flow goes like this: client sends a request to the load balancer’s virtual IP, the load balancer evaluates its routing algorithm and selects a healthy backend based on current load and policy, the request gets forwarded, the server processes it, and the response returns through the load balancer to the client.

The bidirectional proxy model means the load balancer can inspect, modify, and optimize traffic in both directions. Some setups use direct server return where responses bypass the load balancer, but request forwarding always goes through it.

graph TD
    Client1[Client] --> LB[Load Balancer]
    Client2[Client] --> LB
    Client3[Client] --> LB
    LB --> Server1[Server 1]
    LB --> Server2[Server 2]
    LB --> Server3[Server 3]
    Server1 --> LB
    Server2 --> LB
    Server3 --> LB

Layer 4 vs Layer 7 Load Balancing

Network engineers talk about Layer 4 (transport) and Layer 7 (application) load balancing. The layer tells you how deep into the network stack the load balancer inspects when making routing decisions.

Layer 4 Load Balancing

Layer 4 load balancers operate at the transport layer. They route based on source and destination IP addresses plus port numbers, without looking inside the actual request. This makes them faster and able to handle more throughput since parsing happens at a lower level.

Picture L4 as a postal sorter that only looks at the street address, not what is written in the letter. It routes based on network-level information and can process millions of requests per second with minimal latency.

Layer 4 works well for TCP-based protocols like databases or SSH connections, or any protocol where raw throughput matters more than content inspection.

Layer 7 Load Balancing

Layer 7 load balancers operate at the application layer. They can inspect HTTP headers, URLs, cookies, and request bodies. This opens up sophisticated routing based on what the user actually requested.

With L7, you can route based on URL path, send API requests to one cluster and static assets to another, or direct mobile users to a different backend. You can also terminate SSL at the load balancer, inspecting encrypted traffic before forwarding it.

The cost is higher resource usage. Parsing HTTP is more expensive than reading IP and port numbers. Modern L7 balancers are heavily optimized, but this distinction matters when designing systems that need maximum performance.

graph LR
    subgraph "Layer 4"
        L4Req[IP + Port] --> L4Dec[Routing Decision]
    end
    subgraph "Layer 7"
        L7Req[HTTP Headers<br/>URL Path<br/>Cookies] --> L7Dec[Routing Decision]
    end

Software vs Hardware Load Balancers

Back in the day, load balancers were expensive hardware appliances. Companies like F5 sold dedicated network devices for tens of thousands of dollars, with specialized ASICs for packet processing.

The industry shifted toward software. HAProxy, Nginx, and cloud offerings like AWS ALB or Google Cloud Load Balancing commoditized load balancing. You can deploy capable software balancers on commodity hardware or use managed cloud services.

Software wins on flexibility and cost. You can modify routing logic with code changes, integrate with container orchestration, and scale by deploying more instances. Hardware appliance licensing makes less sense in cloud environments.

But hardware still has a place. Regulated industries sometimes require dedicated appliances for compliance. Extremely high-throughput environments, like major video streaming platforms, still use custom ASIC-based solutions. For most web applications though, software approaches work fine.

Sticky Sessions

Session affinity creates problems. If User A logs into Server 1 and their next request goes to Server 2, that server has no memory of the login. The user appears logged out.

Sticky sessions route a particular user’s requests to the same backend server. The load balancer tracks which client maps to which server, using cookies, client IP, or some other identifier.

Cookie-based sticky sessions insert a tracking cookie that identifies the target server. IP-based affinity hashes the client IP to always return the same backend. Header-based approaches use a custom header set by an upstream service.

Sticky sessions cause their own headaches though. They complicate maintenance windows since you cannot take down a server without disconnecting active users. They make horizontal scaling harder because you cannot freely redistribute load. Many applications work better with session state stored in a distributed cache like Redis.

Health Checks and Failover

Load balancers continuously check that backend servers can handle traffic. Health checks run at configurable intervals, testing whether each server responds correctly. A server that fails too many health checks gets marked unhealthy and removed from rotation.

Health checks range from simple TCP connection tests to full HTTP requests with expected response validation. Deeper checks catch more real failures but add latency and load. Most setups use multiple check types: lighter checks more frequently with occasional deeper validation.

graph TD
    LB[Load Balancer] --> HC[Health Checker]
    HC -->|TCP ping| S1[Server 1]
    HC -->|TCP ping| S2[Server 2]
    HC -->|TCP ping| S3[Server 3]
    S1 -->|Healthy| S1Status[✓ Healthy]
    S2 -->|Timeout| S2Status[✗ Unhealthy]
    S3 -->|Healthy| S3Status[✓ Healthy]
    S2Status -->|Remove| Pool[Removed from pool]

When a server fails health checks, the load balancer stops routing traffic to it. Existing connections may be terminated or allowed to drain depending on configuration. Once the server recovers and passes health checks again, it rejoins the pool automatically in most systems.

SSL Termination

Handling HTTPS at the load balancer layer has practical benefits. SSL termination means the load balancer decrypts incoming HTTPS traffic, inspects it if L7, and forwards unencrypted traffic to backend servers. This reduces cryptographic load on application servers.

You manage certificates in one place rather than on every backend. Backend communication can use plain HTTP, reducing CPU overhead on application servers. Your load balancer can inject security headers, rewrite URLs, and perform other transformations on decrypted traffic.

The tradeoff involves trust. Traffic travels unencrypted between load balancer and backend servers, typically within your internal network. In cloud environments, this is usually fine. For sensitive data, you might use SSL passthrough where encrypted traffic flows all the way to backend servers, or re-encrypt before forwarding.

Choosing the Right Load Balancer

Choosing a load balancer depends on your requirements. For simple HTTP traffic with moderate scale, Nginx or HAProxy on a couple of virtual machines works well. They are battle-tested, documented, and free.

Cloud providers offer managed load balancers that integrate with their ecosystems. AWS Application Load Balancer handles L7 routing with rule-based decisions. Network Load Balancer provides ultra-low-latency L4 forwarding for TCP workloads. These services scale automatically and reduce operational overhead.

If you run Kubernetes, the ingress controller often handles load balancing. Options like ingress-nginx, Traefik, or cloud-specific controllers provide L7 routing with tight integration into the container scheduler.

For microservices, service meshes like Istio or Linkerd include load balancing as part of their service-to-service communication layer. These handle traffic shaping, circuit breaking, and retries alongside basic load distribution.

---

When to Use Each Approach

When to Use Load Balancing

Load balancing is essential when:

• You run multiple backend servers serving the same application
• You need high availability (single server failure should not cause outage)
• You want to scale horizontally by adding more servers
• You need to perform maintenance without downtime
• Traffic volume exceeds what a single server can handle
• You want to protect against server overload and failures

When to Use Layer 4 (L4) Load Balancing

L4 is the right choice when:

• You need maximum throughput with minimal latency
• You are load balancing TCP/UDP protocols beyond HTTP
• You do not need to inspect application-layer data
• You are routing database connections or streaming data
• Raw performance matters more than routing intelligence

When to Use Layer 7 (L7) Load Balancing

L7 is the right choice when:

• You need content-based routing (URL path, headers, cookies)
• You want to terminate SSL at the load balancer
• You need to implement sticky sessions
• You are serving multiple applications on the same IP
• You want to rewrite URLs or redirect requests

Global Server Load Balancing

When your application spans multiple geographic regions, simple load balancing breaks down. A user in Tokyo hitting servers in Virginia is a recipe for slow page loads and frustrated users. GSLB solves this by routing users to the closest healthy backend based on geography, server health, and real-time load.

GSLB works at the DNS level. Your DNS responder returns different IP addresses depending on where the query originates. The user gets routed to the nearest healthy region without ever knowing other regions exist.

graph TD
    UserAP[User - APAC] --> DNS[GSLB DNS]
    UserEU[User - EU] --> DNS
    UserUS[User - US] --> DNS
    DNS -->|Return APAC IP| LB_APAC[Load Balancer - Tokyo]
    DNS -->|Return EU IP| LB_EU[Load Balancer - Frankfurt]
    DNS -->|Return US IP| LB_US[Load Balancer - Virginia]

Anycast vs GSLB

One approach worth understanding is Anycast routing. With Anycast, multiple servers share the same IP address, and the network routes packets to the nearest one based on BGP path metrics. This works at the network layer without application involvement.

GSLB gives you more control. You can weigh traffic differently, factor in server health beyond just network reachability, and route based on application-layer signals. Cloud providers offer GSLB as a service: AWS Route 53 geolocation routing, Google Cloud Load Balancing with cloud CDN, and Azure Traffic Manager all fit here.

For globally distributed systems, combining Anycast and GSLB works well. Anycast routes to the nearest region at the network level. GSLB handles fine-grained routing within and between regions based on application health.

Health Checking Across Regions

GSLB health checks must account for entire regions going dark. A health check that only verifies server TCP connectivity misses regional outages. Proper GSLB checks server-level health, regional health, latency thresholds, and capacity limits.

Most GSLB implementations probe from multiple vantage points to distinguish between a slow region and a slow internet path. If three out of five probes fail from different locations, the region gets marked unhealthy.

---

Rate Limiting at the Load Balancer

Load balancers sit in the perfect spot to enforce rate limits. They see every request before it reaches your application, which makes them the first line of defense against abuse, runaway scripts, and intentional DDoS.

Token Bucket Algorithm

The token bucket algorithm is the most common approach. Each client receives a bucket that fills with tokens at a steady rate. Every request consumes a token. When the bucket is empty, requests get rejected or delayed.

# NGINX rate limiting using token bucket
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=100r/s;

server {
    location /api/ {
        limit_req zone=api_limit burst=200 nodelay;
    }
}

Sliding Window Counter

Token bucket allows bursts. If you need more predictable rate limiting, the sliding window counter maintains a rolling count of requests per time window. It uses more memory than token bucket but provides smoother rate limiting.

# HAProxy sliding window configuration
stick-table type ip size 100k expire 60s store http_req_rate(60s)
http-request track-sc0 src table global

Limiting by Different Keys

Rate limits should apply at multiple granularities:

Key	Use Case
Source IP	Single user/clients behind NAT
API Key	Authenticated API consumers
Header (User-Agent)	Bot detection
JWT claim	Per-user service tier limits

Cloud load balancers usually provide built-in rate limiting with automatic IP reputation scoring. AWS ALB has throttling via WAF rules. Google Cloud Load Balancing integrates with Cloud Armor for advanced rate-based policies.

---

Circuit Breaker Pattern Integration

Health checks remove failed backends from rotation. Circuit breakers work one level deeper. They prevent your load balancer from hammering a struggling backend that has not fully failed but is showing signs of strain.

How Circuit Breakers Interact with Load Balancers

A well-designed circuit breaker sits between the load balancer and the backend, monitoring error rates and latency. When a backend starts returning errors or exceeding latency thresholds, the circuit breaker “opens” and trips, immediately returning failures without forwarding the request.

stateDiagram-v2
    Closed --> Open : Error threshold exceeded
    Open --> HalfOpen : Cool-down period elapsed
    HalfOpen --> Closed : Probe request succeeds
    HalfOpen --> Open : Probe request fails

The load balancer still performs health checks on an open circuit breaker. Once the backend recovers and passes enough health checks, the circuit breaker allows traffic through again.

Implementation Approaches

For Kubernetes deployments, service meshes like Istio implement circuit breaking at the sidecar proxy level. You configure outlier detection on your DestinationRule:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: backend-service
spec:
  host: backend-service
  trafficPolicy:
    outlierDetection:
      consecutiveGatewayErrors: 5
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

For traditional deployments, libraries like Hystrix (Java), Resilience4j, or Polly (.NET) implement circuit breakers in your application code. The load balancer health checks still matter — they handle complete failures while circuit breakers handle degraded states.

---

Canary and Blue-Green Deployments

Load balancers make deployment strategies like canary releases and blue-green deployments possible without downtime. Instead of replacing all servers at once, you shift traffic gradually and monitor for problems.

Blue-Green Deployment

Blue-green keeps two identical environments. The current production environment (blue) handles live traffic while the new version (green) sits idle. When you are ready to deploy, you shift all traffic from blue to green at the load balancer level with a single routing rule change.

graph LR
    subgraph "Blue Environment (Current)"
        B1[Server 1]
        B2[Server 2]
    end
    subgraph "Green Environment (New)"
        G1[Server 1']
        G2[Server 2']
    end
    LB[Load Balancer] -->|Route to Blue| B1
    LB -->|Route to Blue| B2
    LB -.->|Ready to switch| G1
    LB -.->|Ready to switch| G2

If something goes wrong, you flip traffic back to blue instantly. The old environment stays warm during the deployment window so rollback never requires rebuilding anything.

Canary Deployment

Canary releases shift a small percentage of traffic to the new version while the rest stays on the current version. You route 5% of users to the new build, monitor error rates and latency, and gradually increase the percentage.

# HAProxy canary configuration
backend canary_backend
    server new_app_1 10.0.1.101:8080 weight 5  # 5% of traffic
    server current_app_1 10.0.1.201:8080 weight 95

# Increase weight gradually as confidence builds
# 5% -> 10% -> 25% -> 50% -> 100%

Canary deployments require more sophisticated monitoring. You need to compare error rates and latency between the canary and production groups. If the canary shows 1% higher error rate while serving 5% of traffic, that is a real signal worth investigating. Load balancers that export metrics to Prometheus or Datadog make this kind of comparison straightforward.

Load Balancer Role in Deployment Safety

The load balancer is the control point for both strategies:

• Instant rollback: Change backend weights to route 100% back to the old version
• Gradual rollout: Shift traffic incrementally to the new version
• Health monitoring integration: Automatically pause rollout if backend error rates spike
• Connection draining: Move users off servers being decommissioned gracefully

Both strategies depend on having enough capacity to run two environments simultaneously. In cloud environments with auto-scaling, this is easier. You spin up green instances alongside blue, shift traffic, then scale down blue.

---

Kubernetes Ingress and Service Mesh Load Balancing

In Kubernetes, load balancing works differently than in traditional setups. The kubelet on each node runs kube-proxy, which sets up IPVS or iptables rules to load balance traffic across pods. But this only handles traffic within the cluster.

Ingress Controllers

For external traffic entering the cluster, Ingress resources define how HTTP/HTTPS routing works. The Ingress controller (like ingress-nginx, Traefik, or cloud-specific controllers) implements those rules and terminates load balancing at the edge.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /users
            pathType: Prefix
            backend:
              service:
                name: users-service
                port:
                  number: 80
          - path: /products
            pathType: Prefix
            backend:
              service:
                name: products-service
                port:
                  number: 80

Ingress controllers handle L7 routing based on host and path, SSL termination, and canary routing through weighted backend services.

Service Mesh Load Balancing

Service meshes like Istio and Linkerd move load balancing from the application layer to the sidecar proxy running alongside each pod. Every outbound request goes through the sidecar, which makes routing decisions based on service mesh policies.

With a service mesh, you get per-request routing, chaos injection for testing, automatic retries with backoff, and fine-grained traffic shaping. Your application code has no idea any of this is happening.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
    - route:
        - destination:
            host: reviews
            subset: v1
          weight: 90
        - destination:
            host: reviews
            subset: v2
          weight: 10

The sidecar proxy handles health checking, load balancing algorithms, and circuit breaking transparently. Your application code has no awareness of which version is handling any given request.

Comparison: Ingress vs Service Mesh

Aspect	Ingress Controller	Service Mesh
Scope	North-south traffic (into cluster)	East-west traffic (within cluster)
Deployment	Cluster-level controller	Per-pod sidecars
Protocol	HTTP/HTTPS primarily	Any L7 protocol
Complexity	Lower	Higher
Use Case	Exposing services externally	Service-to-service communication

Most real deployments end up needing both. The Ingress controller handles traffic entering your cluster from outside. The service mesh handles how those requests reach your application services and how services talk to each other once they are inside.

---

Topic-Specific Deep Dives

When Not to Rely on Load Balancing Alone

Load balancing distributes traffic across servers, but it introduces its own latency and failure modes. Knowing when it falls short helps you design for real-world outages.

Health checks run at intervals, typically every 5-30 seconds. When a backend fails, there is a window where requests still route to the dead server before the health check detects the failure and removes it from the pool. For most applications, this 10-30 second detection window is fine. For systems that need instant failover, you need multi-region active-active setups with DNS-based failover or database-level synchronous replication that handles split-brain scenarios.

Load balancers route requests, they do not synchronize state. If your application maintains server-local state that cannot be externalized to shared storage, taking down a server means losing that state for any user whose requests got routed there. This is why sticky sessions create fragility. Store session state in Redis, Memcached, or a distributed database instead.

Load balancers handle traffic distribution well, but they cannot conjure server capacity out of nowhere. If you expect sudden traffic spikes, you need auto-scaling triggered by load metrics, not reactive scaling after servers are already overwhelmed. Combine load balancing with container orchestration that spins up new instances before demand exceeds capacity. The load balancer then routes to the newly added instances.

Load balancing is necessary but not sufficient for resilient systems. Add proper health checks, external session storage, auto-scaling policies, and circuit breakers that prevent cascading failures when backends degrade.

---

Trade-off Analysis

Layer 4 vs Layer 7 Trade-offs

Aspect	Layer 4	Layer 7
Performance	Higher throughput, lower latency	Higher latency, more CPU usage
Routing Intelligence	IP + port only	Content-based (URL, headers, cookies)
SSL Termination	Not possible (passthrough only)	Full inspection and termination
Protocol Support	Any TCP/UDP protocol	HTTP/HTTPS primarily
Memory Usage	Lower	Higher (stateful inspection)
Complexity	Simpler configuration	More configuration options

Software vs Hardware Trade-offs

Aspect	Software Load Balancers	Hardware Load Balancers
Cost	Low / open source	High ($50K+ appliances)
Flexibility	Easily modified, code changes	Fixed functionality, firmware updates
Scalability	Horizontal (add more VMs)	Vertical (bigger appliance) or clustering
Performance	Good for most web apps	Extreme throughput (ASICs)
Compliance	May not meet regulatory needs	Often compliance-certified
Operations	Requires maintenance	Managed appliance support

Sticky Sessions Trade-offs

Aspect	Without Sticky Sessions	With Sticky Sessions
Load Distribution	Perfect distribution possible	Uneven when clients have different behavior
Session State	Must use shared storage (Redis)	Simpler if server-local state is acceptable
Scaling	Easy horizontal scaling	Harder — cannot freely move clients
Failure Handling	User sessions survive server failure	User loses session if sticky server dies
Complexity	Requires external session store	Simpler initially

Health Check Depth Trade-offs

Check Type	What It Catches	Cost / Trade-off
TCP Connect	Port open, basic reachability	Fast, low overhead, but misses app failures
HTTP Request	Application responding	More accurate, slightly higher latency
Deep HTTP	Response body validation	Catches more failures, highest latency
Custom Script	Business logic verification	Most accurate, operational complexity

Rate Limiting Algorithm Trade-offs

Algorithm	Burst Handling	Memory Usage	Predictability
Token Bucket	Allows bursts	Low	Variable
Sliding Window	No bursts	Higher	Consistent
Fixed Window	Allows bursts	Lowest	Inconsistent at boundaries
Leaky Bucket	Smooths output	Low	Consistent

---

Observability and Security

Metrics

• Request rate (requests per second by backend)
• Response latency (p50, p95, p99 per backend)
• Backend health status (healthy/unhealthy/draining)
• Active connections per backend
• Connection rate (new connections per second)
• SSL handshake rate and latency (if terminating SSL)
• Backend error rate (5xx responses from backends)
• Health check success/failure rate
• Backend response time trends

Logs

• Backend health check failures with details
• SSL handshake failures (certificate errors, protocol mismatches)
• Connection timeouts from backends
• Routing decisions for L7 (which rule matched)
• Backend server added/removed events
• Rate limiting events
• Connection errors and disconnections

Alerts

• Any backend unhealthy for more than 30 seconds
• All backends unhealthy (complete outage)
• Request error rate exceeds threshold
• p99 latency exceeds service level objective
• Active connections approach limits
• Health check failure rate increases
• Unusual traffic patterns (potential attack)

---

Security Checklist

• Restrict access to load balancer management interface
• Use TLS for load balancer to backend communication (internal encryption)
• Implement access controls on health check endpoints
• Monitor for traffic anomalies indicating attack
• Use private VIPs for internal load balancers (not internet-facing)
• Rotate SSL certificates if terminated at load balancer
• Implement rate limiting at load balancer layer
• Log all administrative changes to load balancer config
• Use network ACLs to restrict which clients can reach load balancer
• Enable audit logging for compliance
• Protect against DDoS at load balancer level
• Verify backend servers are not directly accessible (all traffic through LB)

---

Production Failure Scenarios

Failure	Impact	Mitigation
Load balancer itself fails	Complete service outage	Deploy redundant load balancers; use VRRP/keepalived
Backend server fails silently	Requests routed to dead server; errors for users	Implement health checks; remove failed servers quickly
Health check misconfiguration	False positives remove healthy servers	Use multiple check types; set appropriate thresholds
Sticky session overload	One server gets all traffic; cascade failure	Minimize sticky sessions; use session storage (Redis)
SSL termination bottleneck	Load balancer CPU maxes out on encryption	Use SSL offloading hardware; scale horizontally
Connection exhaustion	No new connections accepted; service hangs	Monitor connection counts; implement connection limits
ARP/cache issues with VIP	Traffic routing breaks; intermittent failures	Use keepalived with proper priority; monitor ARP tables
Misconfigured routing rules	Traffic goes to wrong backend; data issues	Test rules in staging; implement gradual rollout

---

Common Pitfalls / Anti-Patterns

Single Point of Failure

A single load balancer is a critical vulnerability. If it fails, no traffic reaches any backend server regardless of how many you have deployed. The backends sit idle while users get connection failures. This failure mode sneaks past monitoring that only tracks backend health.

graph TD
    A[Client] --> B[Single LB]
    B --> C[Server 1]
    B --> D[Server 2]

Load balancers are often deployed as a single instance because they were considered reliable. But software bugs, memory corruption, kernel panics, network partitioning, and operator misconfiguration can all kill a load balancer. Hardware load balancers fail too, sometimes due to power supply issues or fan failures in dense rack-mounted appliances.

Active-passive uses a standby that monitors the active via VRRP. When the active fails, the standby takes over the virtual IP within seconds. keepalived implements this on Linux. Active-active runs multiple load balancers simultaneously, distributing traffic across them for both redundancy and capacity.

Managed cloud load balancers like AWS ALB, Google Cloud Load Balancing, or Azure Load Balancer handle redundancy automatically across availability zones. You get high availability without configuring failover logic. This is the simplest path for cloud deployments. The tradeoff is less control over routing behavior and potential vendor lock-in.

For on-premises deployments, buy a second load balancer instance in active-passive configuration. The cost is trivial compared to the cost of an outage.

Ignoring Health Check Tuning

Health check tuning is often an afterthought, but getting it wrong in either direction causes real problems. Too aggressive and you bounce healthy servers in and out of rotation. Too lenient and failed servers stay in the pool too long, accumulating errors for users.

When health checks run every second with a 1-second timeout and only one failure required to remove a server, transient network hiccups or brief CPU spikes cause unnecessary churn. A server that recovers within 2 seconds still gets removed and must pass several consecutive checks to rejoin. Users see errors during these transitions. Flapping generates load spikes on remaining backends as they absorb traffic from ejected servers.

If you check every 30 seconds and require five failures to remove a server, a backend that died20 seconds ago still receives traffic for the next 130 seconds. During that window, every request to that backend fails. For applications with tight latency budgets or high error rate tolerances, this means extended periods of poor user experience.

A reasonable starting point is 10-second intervals with 3-second timeouts, three failures to remove, and two successes to restore. This gives you detection within 30-40 seconds while avoiding flapping from transient issues. For stateless services with fast startup, you can tighten these. For stateful services where connection disruption is costly, loosen them.

Interval controls how often checks run. Timeout determines how long to wait for a response. Failure threshold sets how many consecutive failures trigger removal. Success threshold determines how many passing checks re-enable a server. Some systems let you configure check types with different weights, running cheap TCP checks frequently and expensive HTTP checks occasionally.

Test your health check configuration under load. A server under heavy request pressure may briefly slow responses enough to trigger aggressive health checks. Monitor your flapping rate and adjust thresholds until you eliminate unnecessary server churn.

Not Planning for Connection Draining

When you need to take a backend server offline for maintenance or deployment, the naive approach is to just stop the server. The load balancer sends new traffic elsewhere, but existing connections break mid-request. Users see errors. If you are doing rolling deployments where you replace servers one by one, this means every server replacement creates a brief outage.

Connection draining allows existing requests on a server being taken offline to complete while preventing new connections from being routed to it. The load balancer stops sending new traffic to the target server but allows in-flight requests to finish. Once all existing connections close or the drain timeout expires, the server can be safely shut down.

Set the drain timeout long enough to handle your worst-case request duration. A request uploading a large file or running a complex database query might take minutes. For most web applications, 60 seconds covers 99% of requests. For APIs with long-running operations, you may need 300 seconds or more. The tradeoff is that during the drain window, your capacity is reduced by the draining server.

NGINX uses the shutdown timeout directive. HAProxy has the timeout drain option. Cloud managed load balancers typically have a configurable drain period, often defaulting to 300 seconds. Kubernetes handles this by default when using rolling updates, giving pods time to complete in-flight requests before terminating them.

Connection draining is essential for zero-downtime deployments. When deploying a new version, you mark the old server as draining. The load balancer routes new traffic to new servers while old servers finish their work. Once drained, the old servers stop and the new servers take over seamlessly. Without draining, each server in a rolling deployment creates a brief outage window.

Skip connection draining and your deployments have visible user impact. Configure it and deployments happen invisibly in the background.

Overusing Sticky Sessions

Sticky sessions seem convenient. Route each user to the same server and their session state stays local. But this convenience comes with hidden costs that compound as your system scales. Most teams that start with sticky sessions eventually migrate away from them.

When requests from the same user always go to the same server, you cannot freely add or remove servers from the pool. If Server 3 is handling 1000 sticky users, removing it disconnects all of them. Your load balancer cannot redistribute those users evenly because their sessions are bound to specific servers. This forces you to keep servers running even when capacity exceeds demand, or to coordinate complex session migration procedures.

Sticky sessions create a scenario where a single server failure affects only the users routed to that server. That sounds like isolation, but it means failures are invisible until they happen. A server approaching memory limits continues receiving requests until it crashes, affecting only its sticky users. Without sticky sessions, a struggling server would naturally receive fewer requests as load redistributes.

Maintenance windows become surgical procedures. You must track which users are on which servers, drain connections carefully, and avoid disrupting active sessions. Rolling deployments require careful sequencing. With hundreds of servers and thousands of users, this coordination becomes error-prone.

Sticky sessions are reasonable for legacy applications where session state cannot be externalized without significant refactoring. They can reduce load on shared session storage for read-heavy workloads where local cache suffices. But even in these cases, treat sticky sessions as a temporary solution, not a permanent architecture.

External session storage works better. Redis provides sub-millisecond read latency for session data. Memcached works well for simple key-value session storage. Your load balancer routes requests freely across all backends, and any backend can serve any user because session state lives in shared storage.

# Problem: All of User A's requests go to Server 1
# If Server 1 fails, User A loses session

# Better: Store sessions in Redis
# All servers can serve User A
session_store: redis

Migrate sessions to Redis first. Once session state is external, disable sticky sessions and watch your load distribution improve automatically.

Not Monitoring Backend Load

A load balancer showing all backends equally utilized is not necessarily distributing load well. Simple metrics like connection count or request count hide the reality that different requests require different amounts of server resources. One backend might be running complex database queries while another handles lightweight static file requests.

Round-robin and least-connections algorithms work at the connection level, not the request level. A backend processing a long-polling request holds a connection for minutes while a backend serving fast static files cycles through many connections per second. Both show similar connection counts, but the long-polling server is much more loaded.

Monitor actual server resource utilization: CPU usage, memory pressure, disk I/O, and database query times. A backend at 80% CPU is more loaded than one at 20% even if both handle the same request rate. Track request latency per backend, not just overall latency. If one backend consistently shows higher latency, it is handling harder requests or is overloaded.

least_conn routes to the backend with fewest active connections, which is a better proxy for actual load on request-response workloads where connection time correlates with work done. weighted_round_robin accounts for different backend capacities by assigning weight to each backend. adaptive algorithms adjust based on real-time CPU or memory metrics, though these require more complex instrumentation.

Watch the gap between load balancer metrics and backend metrics. If backends are at high CPU while load balancer shows balanced traffic, your algorithm is misconfigured or your backends have different capacities. Auto-scaling should trigger on backend resource metrics, not load balancer metrics, to catch these hidden imbalances.

# Simple connection count is not enough
balance roundrobin  # Equal connections, not equal load

# Better: least_conn or weighted by actual load
balance least_conn

Set up dashboards that show backend CPU, memory, and latency alongside load balancer metrics. The moment backend utilization diverges while load balancer metrics look balanced, you have a problem to investigate.

---

Design for Resilience

Design for Redundancy From the Start

Never deploy a single load balancer in production. Use at least two in an active-passive or active-active configuration. For cloud deployments, use the managed load balancer which handles redundancy automatically. The moment you tell yourself “we will add redundancy later” is the moment you guarantee an outage.

Health Checks Are Your Safety Net

Configure health checks to match your actual application requirements. A TCP connect check verifies the port is open but not that the application is working. An HTTP check that validates the response code and optionally the response body catches actual application failures. Set thresholds that avoid flapping. Three failures to remove, two successes to restore is a reasonable starting point.

Prefer L7 When You Need Intelligence

L4 gives you raw performance. L7 gives you control. If you need content-based routing, SSL termination, sticky sessions, or any kind of request inspection, use L7. The performance difference only matters at extreme scale. For most web applications, L7 is the right default.

Distribute Load by Actual Capacity

Round-robin distributes connections evenly, not load. If your backends have different capacities or are running different versions, use weighted distribution. least_conn routes to the backend with the fewest active connections, which is a better proxy for actual load on request-response workloads.

Session and Monitoring

Decouple Session State

Store session state in a distributed cache like Redis rather than relying on sticky sessions. This lets the load balancer do its job properly, distributing requests based on actual load rather than routing constraints. It also means a failed backend does not take user sessions down with it.

Monitor What Matters

Track backend error rates, not just request counts. A backend serving 100 requests per second with 10% errors is worse than one serving 50 requests with 0% errors. Set up alerts on error rate increases before they become outages.

Test Failure Modes Regularly

Run chaos engineering experiments. Terminate a backend server and watch how the load balancer handles it. Verify your monitoring alerts fire. Confirm users recover. Do this in staging regularly enough that you are not surprised when it happens in production.

Operational Excellence

Plan Connection Draining

When taking a backend offline for maintenance, configure connection draining. This allows existing requests to complete while preventing new connections. Set the drain timeout long enough for your worst-case request duration, typically 60 seconds.

Keep Load Balancers Simple

The load balancer should route traffic and enforce policies. Do not try to make it do too much. Offload compression, authentication, and complex business logic to your application servers. Load balancers do one thing well, distributing traffic intelligently.

---

Interview Questions

---

Further Reading

• Load Balancing Algorithms — Deep dive into specific algorithms: round-robin, weighted round-robin, least connections, IP hash, and adaptive load balancing
• CAP Theorem — Fundamental trade-offs in distributed systems that affect load balancing design decisions
• Microservices Architecture — How load balancing fits into larger distributed system patterns
• Service Mesh Deep Dive — Sidecar proxies and mesh-level load balancing for Kubernetes environments
• HAProxy Architecture Guide — Detailed documentation on HAProxy’s internal workings and configuration best practices
• Envoy Proxy Documentation — For understanding modern L7 proxy and service mesh implementation patterns
• AWS Application Load Balancer Documentation — Cloud-native load balancing patterns and integrations

---

Conclusion

Key Bullets

• Load balancers distribute traffic across multiple servers to improve availability and scalability
• L4 operates at transport layer (IP + port) for maximum performance
• L7 operates at application layer (HTTP) for intelligent routing
• Health checks continuously monitor backend availability
• Sticky sessions route users to the same backend but reduce flexibility
• SSL termination at load balancer reduces backend cryptographic load
• Software load balancers (HAProxy, Nginx) work well for most use cases
• Cloud managed load balancers reduce operational overhead
• Redundancy is critical: never have a single load balancer in production

Copy/Paste Checklist

# Check HAProxy backend status (via socket)
echo "show stat" | socat stdio /var/run/haproxy.sock

# Check Nginx upstream status (requires status module)
curl http://localhost:8500/upstream_conf

# Test health check endpoint
curl -I http://backend1:8080/health

# Check active connections
ss -s

# View HAProxy metrics
echo "show info" | socat stdio /var/run/haproxy.sock

# Test backend directly (bypass load balancer)
curl -H "Host: example.com" http://backend1:8080/

---