Break the Link on Purpose: Watching TCP Retransmit, Shrink Its Window, and Still Deliver Every Byte

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

In Lab #7, we watched a TCP connection open and close on a perfect link. Real links drop packets. This lab makes the link lossy on purpose and watches TCP notice the loss and recover.

You will:

• add latency and packet loss to the link with tc netem,
• push a few megabytes and count the retransmitted segments,
• read ss -ti to see RTT, cwnd, and the retransmit counters move,
• see the receive window (win) and window scaling in the capture.

Reading guide: rfc-notes/tcp-retransmission-windowing.md

Prerequisite: TCP Lab 07: One Connection, From SYN to FIN

Expected time: 55–70 minutes.

The Goal

By the end, you should be able to explain this contrast:

What You Will Learn

• Why TCP retransmits: an unacknowledged segment is assumed lost after a timeout (RTO) or duplicate ACKs.
• How RTT drives the retransmission timeout.
• What the receive window advertises and how window scaling extends it.
• How loss slows a transfer without breaking it (reliability on top of an unreliable network).
• How to read retransmit/cwnd/rtt from ss -ti and duplicate segments in a capture.

This lab does not cover:

• Specific congestion-control algorithms (CUBIC, BBR) in depth.
• SACK, ECN, or pacing internals.
• Application-layer behavior on top of TCP.
• Tuning sysctl buffers for throughput.

Where to Read in the RFCs

The Big Picture

Same two nodes as Lab 07. This time we inject delay and loss on the client-side link with tc netem.

client (10.0.0.1) --[ netem: delay 25ms, loss 15% ]-- server (10.0.0.2:8080)

The client sends about 3 MB to the server — first over a clean link, then over the netem-degraded one. We compare the retransmit counts and the elapsed time.

sequenceDiagram
  participant C as client
  participant S as server

  Note over C,S: clean link
  C->>S: seg 1..N (all acked)
  S->>C: ACKs
  Note over C,S: retransmits ~ 0

  Note over C,S: 15% loss
  C->>S: seg k (dropped by netem)
  Note over C: RTO expires / dup ACKs
  C->>S: seg k (retransmit)
  S->>C: ACK
  Note over C,S: slower, but the byte stream is still complete

Both nodes run nicolaka/netshoot (which bundles ip, tc, tcpdump, ss, OpenBSD nc, and socat). No additional images are needed. Since OpenBSD nc lacks --exec/--keep-open, the sink is started with socat instead.

Note: Everything here uses documentation-only address space (RFC 5737), so nothing in this lab touches the real internet.

What You Need

Recommended environment:

• Linux / WSL2 / a Linux VM
• Docker
• containerlab
• tcpdump

Images used:

• nicolaka/netshoot:latest

Running the Lab

The quick path, which deploys, runs the clean transfer, applies netem, captures and measures the lossy transfer, and tears everything down for you:

./scripts/labctl.sh run tcp-08

Or step through it manually:

1. Move into the working directory

cd protocol-lab/examples/tcp-08

2. Deploy and start the sink

sudo containerlab deploy -t tcp-08.clab.yml
docker exec -d clab-tcp-08-server sh -c "socat -u TCP-LISTEN:8080,reuseaddr,fork OPEN:/dev/null"

The server is a sink that discards every byte it receives.

3. Send over the clean link (the baseline)

Note the client's TCP statistics, then send 3 MB.

docker exec clab-tcp-08-client sh -c "cat /proc/net/snmp | grep '^Tcp:'"
time docker exec clab-tcp-08-client sh -c "head -c 3000000 /dev/zero | nc -N -w15 10.0.0.2 8080"
docker exec clab-tcp-08-client sh -c "cat /proc/net/snmp | grep '^Tcp:'"

The second Tcp: line holds the column names and the third the values. The increase in the RetransSegs column is the retransmit count for the clean run — normally close to zero.

4. Make the link lossy

Add delay and loss on the client's eth1.

docker exec clab-tcp-08-client tc qdisc add dev eth1 root netem delay 25ms loss 15%
docker exec clab-tcp-08-client tc qdisc show dev eth1

5. Send again, while capturing

Run a capture in a separate shell.

docker exec -it clab-tcp-08-client tcpdump -i eth1 -n "tcp port 8080"

In yet another shell, peek at the socket statistics during the transfer.

docker exec -it clab-tcp-08-client sh -c "while true; do ss -tino dst 10.0.0.2; sleep 0.5; done"

Then run the transfer.

docker exec clab-tcp-08-client sh -c "cat /proc/net/snmp | grep '^Tcp:'"
time docker exec clab-tcp-08-client sh -c "head -c 3000000 /dev/zero | nc -N -w15 10.0.0.2 8080"
docker exec clab-tcp-08-client sh -c "cat /proc/net/snmp | grep '^Tcp:'"

What to look for:

• The RetransSegs increase is much larger than on the clean run.
• The wall-clock time reported by time is longer — but the transfer still completes.
• ss -tino shows rtt:, cwnd:, retrans:, and rto: side by side. Each loss shrinks cwnd and stretches rto.

6. Remove netem

docker exec clab-tcp-08-client tc qdisc del dev eth1 root

Expected Output

ss -tino dst 10.0.0.2 (during the transfer)

ESTAB 0 197120 10.0.0.1:44210 10.0.0.2:8080
     ... rtt:51.3/12.1 ... cwnd:10 ... retrans:0/23 rto:312 ...

What to look for:

• rtt: sits near 50ms — netem's 25ms in each direction of the round trip.
• The Y in retrans:X/Y is the cumulative retransmit count.
• cwnd: shrinks on loss.

RetransSegs in /proc/net/snmp

• The increase during the lossy transfer is greater than during the clean transfer.

The capture

• wscale (window scale) in the SYN's options.
• win NNNN (the advertised receive window) on every packet after establishment.
• Lines where the same seq appears twice — those are the retransmissions.

Why It Works

TCP promises a reliable byte stream on top of an unreliable network. So whenever "I sent it but no ACK came back," it treats that as loss and sends the data again.

• Retransmission has two triggers. (1) No ACK arrives within a deadline → the RTO timer fires and the segment is retransmitted. (2) The same ACK arrives repeatedly (duplicate ACKs) → the receiver is saying "something after this is missing" → fast retransmit.
• The RTO is built from the RTT (RFC 6298). TCP measures the RTT, tracks its variance too, and sets a timeout with headroom. If the RTT grows, the RTO grows with it.
• The receive window is the amount the receiver advertises it can currently accept. The sender may not have more unacknowledged data in flight than that (flow control). Sixteen bits isn't enough for a modern window, so the window scale option — exchanged in the SYN — extends the effective window (RFC 7323).
• Loss slows things down but doesn't break them. Retransmission fills in the gaps, and congestion control (RFC 5681) slows the sending pace. That's why even at 15% loss, the 3 MB takes longer but arrives complete.

The key observation: TCP detects loss, retransmits, adjusts its pace — and still completes the stream.

Common Pitfalls

• Reading retransmissions as "errors." Retransmission is TCP's normal recovery behavior. It will never be zero on a lossy link.
• Looking at the absolute value of RetransSegs. What matters is the delta relative to the clean run.
• Confusing cwnd with the receive window. cwnd is the sender's internal window, driven by congestion signals. The receive window is what the receiver advertises. The amount you can actually send is the smaller of the two.
• Forgetting to remove netem. The delay and loss will linger into your next experiment. Undo it with tc qdisc del dev eth1 root.
• Setting the loss too high. At 50% or so the transfer effectively stalls. Around 15% is right for observation.
• Expecting tcpdump to label retransmissions. It doesn't. Spot them by the same seq reappearing, or by the retrans counter in ss.

Cleanup

sudo containerlab destroy -t tcp-08.clab.yml --cleanup

If you used labctl.sh run tcp-08, the script runs destroy for you at the end.

Check Your Understanding

1. What are the two triggers that make TCP decide a packet was lost?
2. What is the RTO (retransmission timeout) derived from? What happens to the RTO when the RTT grows?
3. What is the difference between the receive window and the congestion window (cwnd)? Which one determines how much you can actually send?
4. What is the window scale option for? In which packets is it exchanged?
5. Why does a 3 MB transfer still complete even with 15% loss?
6. In ss -tino, what do the X and Y in retrans:X/Y each represent?

References

• RFC 9293: Transmission Control Protocol (TCP)
• RFC 6298: Computing TCP's Retransmission Timer
• RFC 7323: TCP Extensions for High Performance
• RFC 5681: TCP Congestion Control
• RFC 5737: IPv4 Address Blocks Reserved for Documentation
• tc-netem manual page

Verified Run Log (2026-07-05)

This lab has been confirmed reproducible on real hardware.

Environment:

• Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
• Docker 29.1.3
• containerlab 0.77.0
• client / server: nicolaka/netshoot:latest
• The host kernel's sch_netem module was auto-loaded when running tc qdisc add ... netem — no manual modprobe was needed.

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run tcp-08 performed deploy → verify → destroy, and verification.json returned "status": "verified".

Keeping up with environment drift (fixes this verification required)

• netshoot's netcat swap (same as Lab 07). The sink was changed to socat -u TCP-LISTEN:8080,reuseaddr,fork OPEN:/dev/null, and the sender to nc -N -w15 (shutting down the send direction on EOF). Both examples/tcp-08/run.sh and the steps in this article reflect the update.

The lossy link with netem applied

$ docker exec clab-tcp-08-client tc qdisc show dev eth1
qdisc netem 8002: root refcnt 25 limit 1000 delay 25ms loss 15%

Clean vs. lossy retransmit counts (the TcpRetransSegs delta from /proc/net/snmp)

[protocol-lab][tcp-08] clean transfer: 0s, client retransmits=0
[protocol-lab][tcp-08] lossy transfer: 5s, client retransmits=59
[protocol-lab][tcp-08] summary: clean 0s/0 retrans vs lossy 5s/59 retrans

The same 3 MB transfer runs with zero retransmissions and finishes almost instantly on the clean link. With 25ms of delay and 15% loss added, it retransmits 59 times and takes 5 seconds — yet the transfer still completes end to end. That is TCP's reliability at work.

Socket statistics during the transfer (ss -tino) — congestion control in action

# ramp-up (clean-equivalent, cwnd growing)
cubic rtt:25.244/0.431 mss:9448 cwnd:8  ssthresh:10 bytes_sent:404176  bytes_retrans:47240  ... retrans:1/6  lost:1 sacked:5

# loss knocks cwnd and ssthresh down
cubic rtt:25.823/1.219 mss:9448 cwnd:2  ssthresh:2  bytes_sent:3210232 bytes_retrans:510192 ... retrans:2/55 lost:2 sacked:4
cubic rtt:32.396/13.235 mss:9448 cwnd:1 ssthresh:2  bytes_sent:3295264 bytes_retrans:529088 ... retrans:1/57 lost:1

How to read it:

• rtt:25.xxx is the measured RTT reflecting netem's delay 25ms (minrtt:25).
• cwnd (the congestion window) is squeezed from 10 → 8 during ramp-up down to 3 → 2 → 1 each time loss is detected. ssthresh (the slow-start threshold) also drops from 10 → 2. This is RFC 5681's congestion avoidance and loss recovery.
• bytes_retrans, retrans:X/Y, lost, and sacked are the evidence of SACK-based retransmission of the lost segments.
• cubic is the congestion-control algorithm in use.

Cleanup

docker exec clab-tcp-08-client tc qdisc del dev eth1 root
containerlab destroy -t tcp-08.clab.yml --cleanup

That's loss and recovery: TCP notices the missing ACKs, resends exactly what's needed, slows itself down — and the byte stream arrives complete, even through a link dropping one packet in seven.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll go deeper into how TCP decides how fast to send — congestion-control algorithms like CUBIC and BBR, and what their windows look like on the wire.