TCP MSS Clamping: Fixing Path MTU Black Holes by Rewriting the SYN

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

In Lab #25, we watched Path MTU Discovery find a smaller link via ICMP — and saw how it black-holes when that ICMP is filtered. This lab is the operational fix: MSS clamping. A router on the path rewrites the MSS in passing TCP SYNs so both endpoints agree, up front, to send segments that fit the narrowest link — no reliance on PMTUD working.

Reading guide: rfc-notes/mss-clamping.md

Prerequisite: Lab 25: MTU and Path MTU Discovery

Expected time: 35–50 minutes.

The Goal

The client's link is MTU 1500; the r–server link is MTU 1400:

• Without clamping, the client's SYN advertises MSS 1460 (its local 1500 − 40); a full segment is too big for the 1400 link and depends on PMTUD.
• With clamping, r rewrites the SYN's MSS down to 1360 (1400 − 40), so both ends send segments that always fit.

By the end, you should be able to explain this table:

What You Will Learn

• What the TCP MSS option is and how endpoints derive it from local MTU.
• Why the SYN MSS is blind to a smaller link deeper in the path.
• How PMTUD can blackhole when ICMP is filtered (recap of Lab 25).
• How MSS clamping on a router rewrites the SYN's MSS to fit the path.
• Why this matters for tunnels (WireGuard/VXLAN/GRE) that shrink the effective MTU.

This lab does not cover:

• Full PMTUD internals (Lab 25 covers those).
• IPv6 MSS specifics (MSS = MTU − 60 with the larger minimum).
• Per-route MTU pinning or PLPMTUD (RFC 8899).

Where to Read in the RFCs

The Big Picture

r sits between the client and the server, and only the r–server link is MTU 1400.

 client ---- eth1/eth1 ---- r ---- eth2/eth1 ---- server
 MTU 1500                       MTU 1400 (narrow)   MTU 1400
   SYN: mss 1460  --->  r clamps --->  mss 1360 at server

In r's FORWARD chain (mangle table), we clamp the SYN's MSS to the PMTU.

flowchart LR
  C["client (MTU 1500)<br/>SYN mss 1460"] --> R["r<br/>TCPMSS --clamp-mss-to-pmtu"]
  R -->|"SYN rewritten<br/>mss 1360"| S["server (MTU 1400)"]

10.0.0.0/8 is a local, closed range.

Note: Everything here uses local/documentation address space (RFC 1918 / RFC 5737), so nothing in this lab touches the real internet.

What You Need

Recommended environment:

• Linux / WSL2 / a Linux VM
• Docker
• containerlab

Images used:

• nicolaka/netshoot:latest — bundles iptables, tcpdump, curl, and python3.

No additional images are required.

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run mss-37

Or step through it manually:

1. Move into the working directory

cd protocol-lab/examples/mss-37

2. Deploy and start an HTTP server

sudo containerlab deploy -t mss-37.clab.yml
docker exec -d clab-mss-37-server python3 -m http.server 80
docker exec clab-mss-37-r ip -br link show eth2   # mtu 1400

3. Watch the SYN's MSS without clamping

docker exec -d clab-mss-37-server sh -c 'tcpdump -i eth1 -n -c1 "tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0" > /tmp/syn.txt 2>&1'
docker exec clab-mss-37-client curl -s --max-time 4 http://10.0.8.2/ >/dev/null
docker exec clab-mss-37-server grep -oE 'mss [0-9]+' /tmp/syn.txt

You should see mss 1460 — the client's local 1500 − 40.

4. Enable MSS clamping on r

docker exec clab-mss-37-r iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

5. Watch the SYN's MSS again

docker exec -d clab-mss-37-server sh -c 'tcpdump -i eth1 -n -c1 "tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0" > /tmp/syn2.txt 2>&1'
docker exec clab-mss-37-client curl -s --max-time 4 http://10.0.8.2/ >/dev/null
docker exec clab-mss-37-server grep -oE 'mss [0-9]+' /tmp/syn2.txt

Now it's mss 1360 — r clamped it to 1400 − 40.

Expected Output

• r's eth2: mtu 1400.
• Without clamping: the server sees the SYN carrying mss 1460.
• With clamping: the server sees mss 1360.
• The mangle FORWARD chain shows a TCPMSS clamp to PMTU rule.

Why It Works

MSS clamping is "tell both endpoints a safe segment size at the start, instead of relying on PMTUD."

• MSS. The maximum payload TCP will accept in one segment. It is advertised only in the SYN, and the value is local MTU − 40 (IPv4): MTU 1500 → 1460. The sender caps its segments at the peer's advertised MSS.
• The blind spot. Each endpoint only knows its own interface MTU. Even if a narrower link sits deeper in the path (here, the 1400-MTU r–server link), the SYN's MSS doesn't reflect it.
• PMTUD and blackholes (Lab 25). When an oversized segment with DF set hits the narrow link, the router returns an ICMP "fragmentation needed" and the sender shrinks. But if that ICMP is filtered, the sender never learns; large segments keep getting dropped and the connection hangs — a blackhole.
• Clamping. A router on the path rewrites the MSS option in passing SYNs down to the outgoing link's MTU − 40 (if the advertised value is larger). In this lab, r lowers 1460 → 1360. Both ends then send at 1360 or below, so segments always fit the 1400 link — no blackhole even when PMTUD is broken. Rewriting only the SYN is sufficient, because MSS is negotiated only in the SYN.

The key insight: a router on the path injects the path's minimum MTU — invisible to the endpoints — into the SYN's MSS, so both ends send fitting segments from the very first byte. It's a staple in real deployments where tunnels shrink the effective MTU.

Common Pitfalls

• Confusing MSS with MTU. MTU is the whole IP packet; MSS is the TCP payload. In IPv4, MSS = MTU − 40.
• Thinking MSS is renegotiated. It's negotiated only in the SYN, not per data segment — which is exactly why rewriting the SYN's MSS option is all it takes.
• Assuming PMTUD is enough. ICMP filtering can blackhole it. Clamping is the insurance.
• Assuming you can just lower the endpoint's MTU. The narrow link deep in the path is invisible from the endpoint. It's the router on the path that has to do the rewrite.
• Expecting it to help UDP. MSS is a TCP concept; UDP is a different problem.
• The clamp value. --clamp-mss-to-pmtu derives from the outgoing MTU; use --set-mss for a fixed value.

Cleanup

sudo containerlab destroy -t mss-37.clab.yml --cleanup

If you used labctl.sh run mss-37, the script runs destroy for you at the end.

Check Your Understanding

1. What is TCP MSS? How does it differ from MTU (what's the IPv4 relationship)?
2. When is the MSS negotiated? Can it change outside the SYN?
3. Why can't the SYN's MSS reflect a narrower link deeper in the path?
4. When does PMTUD blackhole?
5. What does MSS clamping rewrite, and where? Why does 1460 become 1360 in this lab?
6. Why is clamping so valued with tunnels (WireGuard/VXLAN/GRE)?

References

• RFC 9293: Transmission Control Protocol
• RFC 1191: Path MTU Discovery
• RFC 2923: TCP Problems with Path MTU Discovery
• RFC 5737: IPv4 Address Blocks Reserved for Documentation

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

• Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
• Docker 29.1.3
• containerlab 0.77.0
• client / r / server: nicolaka/netshoot:latest (iptables, tcpdump, curl, python3)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run mss-37 performed deploy → verify → destroy, and verification.json returned "status": "verified".

Without clamping → the SYN carries MSS 1460

Capturing the client's SYN on the server's eth1:

Flags [S], seq ..., options [mss 1460, ...]

That's MSS 1460, derived from the client's link MTU of 1500 — too big for the r–server link (MTU 1400), leaving the connection dependent on PMTUD.

With clamping on r → the SYN carries MSS 1360

mangle FORWARD: TCPMSS  tcp  flags:0x06/0x02  TCPMSS clamp to PMTU

After adding iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu, the same client SYN captured at the server shows:

mss 1360

r rewrote the SYN's MSS option from 1460 → 1360 (1400 − 40) to match its outgoing link (eth2, MTU 1400). From then on both ends send segments of 1360 bytes or less, so everything fits the narrow link — no blackhole, even if ICMP is filtered.

Cleanup

containerlab destroy -t mss-37.clab.yml --cleanup

That's MSS clamping: one iptables rule on the path router, and every TCP connection agrees on a safe segment size before the first byte of data — no working PMTUD required.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll keep digging into how real networks stay reliable when the textbook mechanisms fail.