How traceroute Really Works: TTL, ICMP Time-Exceeded, and Mapping a Path Hop by Hop

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

Every IP packet carries a TTL (time to live) that each router decrements by one. When it reaches zero, the router drops the packet and sends back an ICMP time-exceeded message. traceroute turns this rule into a map: send probes with TTL 1, 2, 3, … and each dying probe reveals the router at that distance.

Reading guide: rfc-notes/traceroute-ttl.md

Prerequisite: TCP Lab 07: Handshake and Teardown (reading captures)

Expected time: 40–55 minutes.

The Goal

This lab builds a real multi-hop path and shows the mechanism:

• client → r1 → r2 → server, with two Linux routers in the middle,
• traceroute from the client lists each hop: 10.0.1.2 (r1), 10.0.2.2 (r2), 10.0.3.2 (server),
• a packet capture shows the ICMP time-exceeded replies (from r1 for TTL 1, from r2 for TTL 2) that traceroute is built on.

By the end, you should be able to explain this table:

What You Will Learn

• What the IP TTL field is for (loop protection) and how routers decrement it.
• What an ICMP time-exceeded message is and who sends it.
• How traceroute uses increasing TTLs to discover each hop.
• Why the hops appear in order, and why the last hop is the destination itself.
• The difference between forwarding (routers) and being an endpoint.

This lab does not cover:

• UDP vs ICMP vs TCP traceroute probe types in depth (we use ICMP mode).
• Load-balanced paths (ECMP) where hops can vary between probes.
• Why some hops show * * * (rate limiting or filtered ICMP) in the real internet.

Where to Read in the RFCs

The Big Picture

Four nodes are chained in series: the client, two routers (r1, r2), and the server.

client ---10.0.1.0/24--- r1 ---10.0.2.0/24--- r2 ---10.0.3.0/24--- server
10.0.1.1             10.0.1.2            10.0.2.2                10.0.3.2
                     10.0.2.1            10.0.3.1

A traceroute from the client to the server reveals the hops in order: r1 → r2 → server. r1 and r2 are plain Linux routers (ip_forward plus static routes).

sequenceDiagram
  participant C as client
  participant R1 as r1
  participant R2 as r2
  participant S as server

  C->>R1: probe TTL=1
  Note over R1: TTL 1->0, drop
  R1-->>C: ICMP time-exceeded (from 10.0.1.2)
  C->>R1: probe TTL=2
  R1->>R2: TTL 2->1, forward
  Note over R2: TTL 1->0, drop
  R2-->>C: ICMP time-exceeded (from 10.0.2.2)
  C->>R1: probe TTL=3
  R1->>R2: forward
  R2->>S: forward
  S-->>C: reply (destination reached)

Note: The 10.0.0.0/8 subnets used here are local and closed, so nothing in this lab touches the real internet.

What You Need

Recommended environment:

• Linux / WSL2 / a Linux VM
• Docker
• containerlab

Images used:

• nicolaka/netshoot:latest — bundles traceroute, ping, and tcpdump.

No additional images are required. The routers are just Linux with ip_forward and static routes (which run.sh sets up).

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run trace-19

Or step through it manually:

1. Move into the working directory

cd protocol-lab/examples/trace-19

2. Deploy and set up the routes

sudo containerlab deploy -t trace-19.clab.yml
# Enable forwarding on r1/r2 and add static routes on each node (run.sh does this)
docker exec clab-trace-19-r1 sysctl -w net.ipv4.ip_forward=1
docker exec clab-trace-19-r2 sysctl -w net.ipv4.ip_forward=1
docker exec clab-trace-19-client sh -c "ip route add 10.0.2.0/24 via 10.0.1.2; ip route add 10.0.3.0/24 via 10.0.1.2"
docker exec clab-trace-19-r1 ip route add 10.0.3.0/24 via 10.0.2.2
docker exec clab-trace-19-r2 ip route add 10.0.1.0/24 via 10.0.2.1
docker exec clab-trace-19-server sh -c "ip route add 10.0.2.0/24 via 10.0.3.1; ip route add 10.0.1.0/24 via 10.0.3.1"

3. Trace the path

docker exec clab-trace-19-client traceroute -I -n 10.0.3.2

What to look for:

 1  10.0.1.2   ...   <- r1
 2  10.0.2.2   ...   <- r2
 3  10.0.3.2   ...   <- server (destination)

Each line is one hop. By raising the TTL one step at a time, traceroute pulls out routers at increasing distances, in order.

4. Confirm the TTL mechanism with a capture

docker exec -d clab-trace-19-client tcpdump -i eth1 -n -w /tmp/te.pcap "icmp"
docker exec clab-trace-19-client ping -c1 -t1 10.0.3.2   # TTL 1: dies at r1
docker exec clab-trace-19-client ping -c1 -t2 10.0.3.2   # TTL 2: dies at r2
docker exec clab-trace-19-client pkill -INT tcpdump
docker exec clab-trace-19-client tcpdump -n -vv -r /tmp/te.pcap

You'll see 10.0.1.2 > 10.0.1.1: ICMP time exceeded in-transit (from r1) and 10.0.2.2 > 10.0.1.1: ICMP time exceeded (from r2). These messages are exactly what builds each hop line in traceroute's output.

Expected Output

• traceroute: hop 1 = 10.0.1.2 (r1), hop 2 = 10.0.2.2 (r2), hop 3 = 10.0.3.2 (server).
• Capture: ICMP time exceeded messages from r1 (10.0.1.2) and r2 (10.0.2.2).

Why It Works

TTL (time to live) exists as insurance against packets looping forever. Every router decrements it by one when forwarding, and drops the packet when it hits zero. traceroute cleverly exploits this side effect.

• The TTL decrement. A router decrements TTL before forwarding a packet. If the result is 0, it drops the packet instead of forwarding it and returns an ICMP time-exceeded (RFC 792) to the sender. The source address of that ICMP message is what reveals "which router that was."
• The traceroute trick. Send a probe toward the destination with TTL=1: it hits 0 at the first router (r1), which returns a time-exceeded — hop 1 identified. Next, TTL=2 dies at r2 — hop 2. TTL=3 reaches the server — the end of the path. One router at a time, one per distance, the path gets pulled out.
• Ordering and the endpoint. The higher the TTL, the farther the router that answers — so hops always appear nearest-first. The last hop is the destination itself, which returns a normal reply rather than a time-exceeded.
• Forwarding vs endpoint. r1 and r2 play the "pass it along" role (ip_forward); the client and server are endpoints. Decrementing TTL and emitting time-exceeded is the forwarder's — the router's — job.

The key insight: a simple loop-prevention field, TTL, becomes a tool that makes the path visible one hop at a time.

Common Pitfalls

• Confusing TTL with hop count. TTL is an upper bound on remaining hops. traceroute increments it 1, 2, 3, … to draw out each hop.
• Who sends time-exceeded. The router that decremented TTL to 0 — not the destination (the destination replies normally).
• Lines of * * *. On the real internet, some routers rate-limit or drop ICMP, so hops can be invisible. This lab is a closed environment, so everything shows.
• Probe types. traceroute can use UDP, ICMP, or TCP probes. This lab uses -I (ICMP). Results can look different depending on the environment.
• Asymmetric paths. The forward and return paths can differ. traceroute shows the forward path.
• Missing static routes. In a multi-hop setup, every node also needs a return route. Miss one, and things stall midway.

Cleanup

sudo containerlab destroy -t trace-19.clab.yml --cleanup

If you used labctl.sh run trace-19, the script runs destroy for you at the end.

Check Your Understanding

1. What is the IP TTL field originally for? How does a router handle TTL?
2. When TTL reaches 0, what does the router send back? What does its source address tell you?
3. How does traceroute draw out each hop, one router at a time?
4. Why is traceroute's final hop not a time-exceeded message?
5. Why can hops show up as * * * on the real internet?
6. What is the difference between a forwarding node and an endpoint node? Which is which in this lab?

References

• RFC 791: Internet Protocol (Time to Live)
• RFC 792: Internet Control Message Protocol (Time Exceeded)
• RFC 1812: Requirements for IP Version 4 Routers
• RFC 5737: IPv4 Address Blocks Reserved for Documentation
• traceroute manual page

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

• Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
• Docker 29.1.3
• containerlab 0.77.0
• client / r1 / r2 / server: nicolaka/netshoot:latest (traceroute, tcpdump included)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run trace-19 performed deploy → verify → destroy, and verification.json returned "status": "verified".

traceroute maps the path hop by hop

$ docker exec clab-trace-19-client traceroute -I -n 10.0.3.2
traceroute to 10.0.3.2 (10.0.3.2), 30 hops max, 46 byte packets
 1  10.0.1.2  0.003 ms     <- r1
 2  10.0.2.2  0.001 ms     <- r2
 3  10.0.3.2  0.000 ms     <- server (destination)

The TTL mechanism: ICMP time-exceeded comes back from each router

$ docker exec clab-trace-19-client tcpdump -n -vv -r icmp.pcap
10.0.1.2 > 10.0.1.1: ICMP time exceeded in-transit, length 92   <- TTL 1 dies at r1
10.0.2.2 > 10.0.1.1: ICMP time exceeded in-transit, length 92   <- TTL 2 dies at r2

The TTL=1 probe hits 0 at the first router, r1 (10.0.1.2), and r1 returns a time-exceeded. The TTL=2 probe dies at r2 (10.0.2.2). traceroute maps the path by exploiting exactly this "one router answers per distance" behavior — a simple loop-prevention field turned into a path-visualization tool.

Cleanup

containerlab destroy -t trace-19.clab.yml --cleanup

That's traceroute: no dedicated discovery protocol, just TTL doing its loop-prevention job and every router along the way announcing itself as it drops your probe.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll keep pulling packets apart — from mapping paths to watching what protocols do at each end of them.