L4 Load Balancing with IPVS: One VIP, Three Backends, and a Perfect 10/10/10 Split

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

This is the third way to spread load across servers. Anycast (Lab 31) let routing pick one instance; ECMP (Lab 32) let routing hash flows across links. Here, a load balancer actively distributes connections to a pool.

Reading guide: rfc-notes/l4-load-balancing.md

Prerequisite: Lab 20: NAT — Source Address Translation

Expected time: 40–55 minutes.

The Goal

One virtual IP (10.0.9.100) fronts three backends. A Linux IPVS director schedules each incoming connection round-robin and NATs it to a real backend (and NATs the reply back), so the client only ever talks to the VIP:

• the client connects to http://10.0.9.100/ thirty times,
• IPVS sends the connections to backend1, backend2, backend3, backend1, … in turn,
• the responses cycle through all three — 10 / 10 / 10 — while the client never sees a backend address.

By the end, you should be able to explain this table:

What You Will Learn

• What a VIP + real-server pool is, and how clients see only the VIP.
• How an L4 load balancer distributes connections (not HTTP requests) by a scheduler (round-robin here).
• How IPVS NAT mode works, and why backends must default-route through the director.
• Why the director is stateful (a connection table) so replies rewrite back to the VIP.
• How L4 LB differs from anycast (31) and ECMP (32), and from an L7 reverse proxy.

This lab does not cover:

• L7 (HTTP-aware) load balancing — routing by URL/Cookie/host.
• Health checking / failover of dead backends (keepalived).
• DR/TUN forwarding modes, or persistence/session affinity in depth.

Where to Read in the RFCs

The Big Picture

The topology is client — lb (VIP + IPVS) — sw (bridge) — backend1/2/3.

 client            lb (director)              backend1 (10.0.10.11)
 10.0.9.2 --- eth1 --+ VIP 10.0.9.100    +--- backend2 (10.0.10.12)
                     + eth2 10.0.10.1 -- sw --+ backend3 (10.0.10.13)
                       IPVS rr (NAT)          (default gw = lb)

The client connects to the VIP. For each connection, IPVS picks a backend round-robin and forwards it with NAT. The backends' replies flow back through their default gateway (the lb), where the source is rewritten back to the VIP.

flowchart LR
  C["client<br/>curl http://10.0.9.100/ ×30"] --> V["lb: IPVS VIP 10.0.9.100<br/>scheduler = round-robin"]
  V -->|conn 1,4,…| B1["backend1 → 'backend1'"]
  V -->|conn 2,5,…| B2["backend2 → 'backend2'"]
  V -->|conn 3,6,…| B3["backend3 → 'backend3'"]

10.0.9.0/24 (client side) and 10.0.10.0/24 (backend side) are local, closed ranges.

Note: Everything here uses local/documentation address space (RFC 1918 / RFC 5737), so nothing in this lab touches the real internet.

What You Need

Recommended environment:

• Linux / WSL2 / a Linux VM (the IPVS kernel module ip_vs must be available)
• Docker
• containerlab

Images used:

• nicolaka/netshoot:latest — bundles ipvsadm, curl, and python3.

No additional images are required.

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run lb-33

Or step through it manually:

1. Move into the working directory

cd protocol-lab/examples/lb-33

2. Deploy and start the backend responders

sudo containerlab deploy -t lb-33.clab.yml
docker exec -d clab-lb-33-backend1 python3 /responder.py backend1
docker exec -d clab-lb-33-backend2 python3 /responder.py backend2
docker exec -d clab-lb-33-backend3 python3 /responder.py backend3

Each backend runs a tiny HTTP responder on 0.0.0.0:80 that returns its own name.

3. Configure IPVS (VIP, round-robin, NAT)

docker exec clab-lb-33-lb ipvsadm -A -t 10.0.9.100:80 -s rr
docker exec clab-lb-33-lb ipvsadm -a -t 10.0.9.100:80 -r 10.0.10.11:80 -m
docker exec clab-lb-33-lb ipvsadm -a -t 10.0.9.100:80 -r 10.0.10.12:80 -m
docker exec clab-lb-33-lb ipvsadm -a -t 10.0.9.100:80 -r 10.0.10.13:80 -m
docker exec clab-lb-33-lb ipvsadm -L -n

-s rr selects the round-robin scheduler; -m means masquerading (NAT) forwarding.

4. Hit the VIP repeatedly

docker exec clab-lb-33-client sh -c 'for i in $(seq 1 6); do curl -s http://10.0.9.100/; done'

The responses cycle backend1 → backend2 → backend3 → … while the VIP never changes.

5. Count the distribution

docker exec clab-lb-33-client sh -c 'for i in $(seq 1 30); do curl -s http://10.0.9.100/; done' | sort | uniq -c

The 30 requests land almost exactly evenly across the three backends (10 / 10 / 10).

Expected Output

• ipvsadm -L -n: the VIP 10.0.9.100:80 rr with three real servers below it (Masq).
• The response sequence cycles through backend1/2/3 in round-robin order.
• The 30-request distribution is 10 / 10 / 10 (round-robin, so perfectly even).
• The client's destination is always the VIP — backend addresses are never visible.

Why It Works

An L4 load balancer is "one VIP, a pool of real servers, and a director that assigns each connection to one of them."

• VIP and pool. The client connects to the VIP (10.0.9.100). The director forwards that connection to a real server (10.0.10.11, etc.). The servers behind it are invisible to the client, so you can grow or shrink the pool without touching any client.
• Scheduler. The policy for which backend gets the next connection. Here it's round-robin (cycle through them in order); others include least-connection, weighted variants, and source-hashing. Because this is L4, it never looks inside HTTP (URL/Cookie) — it only sees the 5-tuple. Routing on content is the job of an L7 LB, which is a different beast.
• NAT mode. The director rewrites the connection's destination to the backend (DNAT) and rewrites the reply's source back to the VIP. That's why replies must pass through the director — and why each backend's default gateway is set to the director.
• Stateful. The director keeps a connection table that maps every packet of a connection to the same backend and maps replies back to the VIP. This is the decisive difference from anycast/ECMP (which scatter traffic statelessly via routing/hashing) — an LB holds state to keep both directions consistent.
• Closing the trilogy. Anycast (31): routing picks one instance. ECMP (32): routing hashes flows across links. LB (33): a director actively distributes connections to a pool.

The key insight: behind a single virtual address, the director picks a backend per connection and brokers the round trip with NAT.

Common Pitfalls

• Thinking L4 looks at HTTP. It doesn't. Routing by URL/Cookie is L7 LB territory; L4 just forwards connections.
• Thinking the VIP lives on the backends. In NAT mode the VIP is on the director. Backends have their real IPs.
• The return path. In NAT mode, if the backends' default gateway isn't the director, replies never get rewritten back to the VIP and connections fail to establish.
• Assuming health checking is automatic. Bare IPVS keeps rotating dead backends into the mix. Health is the job of keepalived and friends (out of scope here).
• Assuming stickiness. Round-robin spreads per connection. To pin a client to one backend, use source-hashing or persistence.
• The ip_vs module. The host needs the IPVS kernel module (it auto-loads when ipvsadm runs).

Cleanup

sudo containerlab destroy -t lb-33.clab.yml --cleanup

If you used labctl.sh run lb-33, the script runs destroy for you at the end.

Check Your Understanding

1. What are an L4 load balancer's VIP and real-server pool? What does the client see?
2. How does an L4 LB choose where to send a connection? Does it look at HTTP content?
3. In IPVS NAT mode, why must the backends' default gateway be the director?
4. What does it mean for the director to be "stateful"? Why is the state necessary?
5. Contrast anycast (31), ECMP (32), and LB (33) by "what decides the distribution" and "whether state is kept."
6. What's the difference between an L4 LB and an L7 reverse proxy?

References

• Linux Virtual Server (IPVS) documentation
• RFC 2663: IP Network Address Translator (NAT) Terminology and Considerations
• RFC 7424: Mechanisms for Optimizing LAG/ECMP Component Link Utilization
• RFC 5737: IPv4 Address Blocks Reserved for Documentation

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

• Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
• Docker 29.1.3
• containerlab 0.77.0
• client / lb / backend1–3 / sw: nicolaka/netshoot:latest (ipvsadm, curl, python3)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run lb-33 performed deploy → verify → destroy, and verification.json returned "status": "verified".

The IPVS configuration (VIP, round-robin, NAT)

TCP  10.0.9.100:80 rr
  -> 10.0.10.11:80                Masq    1      0          0
  -> 10.0.10.12:80                Masq    1      0          0
  -> 10.0.10.13:80                Masq    1      0          0

The VIP 10.0.9.100:80 is registered with the round-robin scheduler and three real servers in masquerading (NAT) mode.

30 requests split evenly across 3 backends

The client connected to http://10.0.9.100/ (the same VIP every time) thirty times:

first 6 responses: backend2 backend1 backend3 backend2 backend1 backend3
distribution over 30 requests:
     10 backend1
     10 backend2
     10 backend3

Each connection went to the next backend in round-robin order, and the 30 requests split cleanly into 10 / 10 / 10. The client's destination was the VIP 10.0.9.100 throughout — no backend address was ever visible. The director brokered every connection via NAT.

(Note: the packet/byte counters in ipvsadm -L -n --stats can show 0 in this environment (via nsenter), but the actual distribution is confirmed by the client-side response counts of 10/10/10.)

Cleanup

containerlab destroy -t lb-33.clab.yml --cleanup

That's L4 load balancing: one virtual address up front, a director choosing a backend per connection, and NAT quietly stitching both directions together — the stateful counterpart to anycast and ECMP.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll climb the stack and look at what an L7 reverse proxy can do that an L4 director can't — routing requests by what's inside them.