PathVector Blog

Anycast Explained: One IP Address, Many Servers, and BGP Picks the Winner

Announce the same IP from two servers, let BGP choose the best path, then kill the winner and watch traffic fail over automatically — no client changes needed.

  • bgp
  • network
  • containerlab
  • anycast

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

In Lab #29, multicast sent a single packet to a group. Anycast pulls the opposite trick: the same address lives on several servers, and the routing system quietly steers each client to one of them. Nothing in the packet asks for the "nearest" server — BGP already knows one best path to that address, and the packet just follows it.

Reading guide: rfc-notes/anycast.md

Prerequisite: BGP Lab 01: Announcing a Prefix over eBGP

Expected time: 45–60 minutes.

The Goal

Two servers both announce 10.0.0.100/32 into BGP:

  • server-b prepends its own AS (making a longer AS_PATH), so r1 prefers server-a.
  • The client fetches http://10.0.0.100/ and gets server-a.
  • Then server-a's uplink fails. BGP withdraws its route and r1 reconverges onto server-b.
  • The client fetches the same http://10.0.0.100/ and now gets server-b — automatic failover, with no change on the client.

By the end, you should be able to explain this table:

State r1's best path to 10.0.0.100 client gets
both up via server-a (AS_PATH 65001, shorter) server-a
server-a down via server-b (AS_PATH 65002 65002 65002) server-b

What You Will Learn

  • What anycast is: one prefix announced from many places, with routing installing exactly one best path.
  • Why "nearest" means nearest in routing terms (AS_PATH length here), not geographic distance.
  • How BGP best-path selection turns two identical announcements into one installed route.
  • How failover works: withdraw the winner's route, and routing reconverges automatically.
  • Where anycast is used in the real world (root DNS, 1.1.1.1 / 8.8.8.8, CDNs, DDoS absorption) and its caveats for stateful traffic.

This lab does not cover:

  • Fine-grained load balancing across instances — anycast pins a client to one.
  • Stateful anycast (session sync, consistent hashing) for long-lived TCP.
  • IGP-based anycast (OSPF/IS-IS metrics) — here we use eBGP AS_PATH.

Where to Read in the RFCs

Reference What to focus on
RFC 4271 §9.1 BGP best-path selection (choosing one route by AS_PATH length)
RFC 4786 Operation of anycast services (multiple announcements, catchment, switchover)
RFC 7094 The definition of anycast and its caveats for stateful traffic
RFC 5737 / RFC 1918 Confirming the addresses used here are local/documentation-only

The Big Picture

Behind the client sits r1 (AS 65000). r1 connects over eBGP to both server-a (AS 65001) and server-b (AS 65002). Both servers announce the same VIP.

 client            r1 (AS 65000)          server-a (AS 65001)  lo 10.0.0.100/32
 10.0.9.2 --- eth1 ---+--- eth2 --- 10.0.1.0/30 --- (BGP: network 10.0.0.100/32)
                      |
                      +--- eth3 --- 10.0.2.0/30 --- server-b (AS 65002)  lo 10.0.0.100/32
                                    (BGP: network 10.0.0.100/32, prepend 65002 65002)

r1 receives two paths to the VIP and picks server-a, whose AS_PATH is shorter, as best. When server-a fails, it reconverges onto server-b.

flowchart LR
  C["client<br/>wget http://10.0.0.100/"] --> R["r1 (AS 65000)<br/>best path = server-a"]
  R -->|"best: AS_PATH 65001"| A["server-a<br/>lo 10.0.0.100/32<br/>→ 'server-a'"]
  R -.->|"standby: 65002 65002 65002<br/>(prepended, longer)"| B["server-b<br/>lo 10.0.0.100/32<br/>→ 'server-b'"]

10.0.0.0/8 is a local, closed range. The VIP is 10.0.0.100.

Note: Everything here uses local/documentation address space (RFC 1918 / RFC 5737), so nothing in this lab touches the real internet.

What You Need

Recommended environment:

  • Linux / WSL2 / a Linux VM
  • Docker
  • containerlab

Images used:

  • frrouting/frr:latest — BGP on r1, server-a, and server-b. It bundles python3, which the HTTP responder uses.
  • nicolaka/netshoot:latest — the client, providing wget and traceroute.

No additional images are required.

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run anycast-31

Or step through it manually:

1. Move into the working directory

cd protocol-lab/examples/anycast-31

2. Deploy

sudo containerlab deploy -t anycast-31.clab.yml

Both servers carry 10.0.0.100/32 (the VIP) on lo and announce it with FRR's network 10.0.0.100/32. server-b prepends its own AS via an outbound route-map.

3. Start the identity responders

docker exec -d clab-anycast-31-server-a python3 /responder.py server-a
docker exec -d clab-anycast-31-server-b python3 /responder.py server-b

Each server runs a tiny HTTP responder on 0.0.0.0:80 that returns its own name — and it answers on the VIP too.

4. Confirm BGP's best path (server-a wins)

docker exec clab-anycast-31-r1 vtysh -c "show bgp ipv4 unicast 10.0.0.100/32"
docker exec clab-anycast-31-r1 ip route get 10.0.0.100     # via 10.0.1.2 (server-a)

You should see two paths, with the shorter-AS_PATH server-a (65001) chosen as best.

5. Fetch the VIP from the client (server-a answers)

docker exec clab-anycast-31-client wget -qO- http://10.0.0.100/     # → server-a
docker exec clab-anycast-31-client traceroute -n 10.0.0.100

6. Trigger failover: drop server-a's link

docker exec clab-anycast-31-server-a ip link set eth1 down
sleep 5
docker exec clab-anycast-31-r1 ip route get 10.0.0.100     # via 10.0.2.2 (server-b)
docker exec clab-anycast-31-client wget -qO- http://10.0.0.100/     # → server-b

Same VIP, but the response now comes from server-b.

7. Restore it

docker exec clab-anycast-31-server-a ip link set eth1 up

Expected Output

  • show bgp ... 10.0.0.100/32: two paths, with the shorter-AS_PATH server-a as best.
  • ip route get 10.0.0.100: via 10.0.1.2 (server-a) before the failure, via 10.0.2.2 (server-b) after.
  • wget http://10.0.0.100/: server-a before the failure, server-b after — and the destination IP never changes.

Why It Works

Anycast is "one address, many instances, routing decides." The same prefix (here 10.0.0.100/32) is announced into routing from multiple nodes, and each router installs exactly one best path to that prefix in its FIB. So every client behind a given router always reaches one instance. The sender does nothing special — the destination is just a single IP.

  • Best-path selection. r1 receives two paths to the VIP and BGP picks one as best. Here the deciding factor is AS_PATH length: server-a is 65001 (length 1), while server-b, after prepending, is 65002 65002 65002 (length 3). The shorter path — server-a — wins. That's why "nearest" means nearest in routing-metric terms, not physical distance.
  • Catchment. Which clients land on which instance is determined by the routing topology. In production this often correlates with geography, but it's BGP/IGP that actually decides.
  • Failover. When server-a's link drops, the BGP session between r1 and server-a tears down and server-a's route is invalidated. r1 selects the remaining server-b as best and updates its FIB (reconvergence). The destination address stays the same; traffic simply flows to server-b. No client reconfiguration is needed.
  • Why it's useful. Root DNS, 1.1.1.1 / 8.8.8.8, and CDNs serve the same IP from many instances worldwide. Clients land on a nearby one for low latency, traffic shifts automatically if one dies, and attacks get spread out and absorbed.

The key insight: just announcing the same prefix from multiple places lets ordinary routing handle both "selection" and "failover." No dedicated protocol required.

Common Pitfalls

  • Confusing anycast with a load balancer. A single client basically pins to one instance. Distribution emerges because many clients each have a different best path. Fine-grained load balancing is a separate mechanism.
  • Thinking "nearest" means geographic. It's nearest in AS_PATH / IGP-metric terms. Adjusting the number of prepends changes the preference.
  • Moving stateful traffic. If the best path changes mid–long-lived TCP, packets can jump to a different instance and break the connection. Classically, anycast suits DNS (UDP); CDNs use it for HTTP too, relying on stable convergence.
  • Using the VIP as the router-id. Keep FRR's router-id distinct from the VIP (this lab uses 10.11.11.11 / 10.22.22.22).
  • Return path. Servers need a route back to the client subnet (this lab points a default route at r1).
  • Convergence time. Failover isn't instant. Detecting the BGP session drop and recomputing takes a few seconds (about 2 seconds in this environment).

Cleanup

sudo containerlab destroy -t anycast-31.clab.yml --cleanup

If you used labctl.sh run anycast-31, the script runs destroy for you at the end.

Check Your Understanding

  1. What is anycast? How does it differ from unicast and multicast?
  2. When two servers announce the same 10.0.0.100/32, why does r1 use only one of them?
  3. Why does server-a win in this lab? What is server-b doing?
  4. "Nearest" in what sense? How can it diverge from geographic distance?
  5. When server-a goes down, why can the same VIP still be answered by server-b? What is happening under the hood?
  6. Why is anycast a good fit for DNS but something to be careful with for long-lived TCP?

References

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

  • Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
  • Docker 29.1.3
  • containerlab 0.77.0
  • r1 / server-a / server-b: frrouting/frr:latest (BGP + python3 responder)
  • client: nicolaka/netshoot:latest (wget, traceroute)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run anycast-31 performed deploy → verify → destroy, and verification.json returned "status": "verified".

The same VIP: server-a before the failure, server-b after

[protocol-lab][anycast-31] r1 route to 10.0.0.100 is via 10.0.1.2 (after 8s)
[protocol-lab][anycast-31] before failover: server-a server-a server-a
[protocol-lab][anycast-31] + docker exec clab-anycast-31-server-a ip link set eth1 down
[protocol-lab][anycast-31] r1 route to 10.0.0.100 is via 10.0.2.2 (after 2s)
[protocol-lab][anycast-31] after failover: server-b server-b server-b

Throughout, the client only ever hits http://10.0.0.100/ — the same VIP. Before the failure, all three requests return server-a; after server-a's link goes down, routing reconverges in about 2 seconds and all three then return server-b. The destination address is never changed.

r1's BGP best-path tells the "selection" and "failover" story

# before the failure — two paths; the shorter-AS_PATH server-a is best
Paths: (2 available, best #2, table default)
  65002 65002 65002        <- server-b (longer, due to prepend)
    10.0.2.2 from 10.0.2.2 (10.22.22.22)
  65001                    <- server-a (shorter)
    10.0.1.2 from 10.0.1.2 (10.11.11.11)
      Origin IGP, metric 0, valid, external, best (AS Path)

# after the failure — server-a's path is gone; the remaining server-b is best
Paths: (1 available, best #1, table default)
  65002 65002 65002
    10.0.2.2 from 10.0.2.2 (10.22.22.22)
      Origin IGP, metric 0, valid, external, best (First path received)
  • Before the failure, server-a (65001) is chosen for the reason best (AS Path). server-b is 65002 65002 65002 (length 3) after prepending, so it's the non-preferred standby.
  • When server-a's link drops, the BGP session goes down and that route is withdrawn. r1 reselects the remaining server-b as best and updates its FIB to via 10.0.2.2. There's no dedicated switchover mechanism — ordinary BGP reconvergence is the failover.

traceroute 10.0.0.100 (before the failure) showed the two hops client → 10.0.9.1 (r1) → 10.0.0.100, confirming the packet reaches the VIP directly.

Cleanup

containerlab destroy -t anycast-31.clab.yml --cleanup

That's anycast: one prefix announced from many places, and plain old routing doing both the picking and the failover for you — no special protocol involved.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll dig deeper into BGP path attributes and how operators steer traffic beyond a simple AS_PATH prepend.