MTU and fragmentation for UDP shred delivery

MTU is a per-hop packet limit

Maximum transmission unit, or MTU, is the largest network-layer packet an interface can send without fragmentation on that link. Standard Ethernet interfaces commonly expose an IP MTU of 1,500 bytes. That number covers the IPv4 or IPv6 packet, including its IP header, but not the Ethernet header, frame check sequence, preamble, or inter-frame gap.

A path can contain links with different MTUs. The path MTU is the smallest usable IP MTU between endpoints. A sender configured for 1,500 does not prove that every tunnel or provider segment supports 1,500. Overlays consume bytes for outer headers and may reduce the inner MTU exposed to workloads.

UDP does not split application records across datagrams. The application passes one payload to sendmsg. IP then decides whether the resulting datagram fits, can be fragmented, or must fail. A raw feed should size each datagram below the intended path MTU with room for IP and UDP headers. That keeps one application packet equal to one IP packet under ordinary delivery.

Header arithmetic sets the payload ceiling

An IPv4 header is normally 20 bytes without options, and a UDP header is 8 bytes. Under a 1,500 byte IP MTU, the largest unfragmented UDP payload is therefore 1,472 bytes. IPv6 has a fixed 40 byte base header, giving a 1,452 byte UDP payload before any IPv6 extension headers.

The formula is direct:

maximum UDP payload = path MTU - IP header - UDP header

With IPv4 and a 1,400 byte inner MTU, the payload ceiling is 1,372 bytes. With IPv6 it is 1,352 bytes. A 1,216 byte mean shredstream.sh packet and a 1,228 byte maximum remain below those examples if the product size is the UDP payload. Packet captures should confirm which layer a local length field represents before performing an incident calculation.

VLAN tags usually do not reduce the configured IP MTU when the Ethernet network supports the added frame size, but hardware and provider limits still matter. Encapsulation is the larger risk because inner and outer packets coexist on the constrained link.

IPv4 fragmentation splits one datagram

IPv4 routers may fragment a packet when it exceeds an outgoing MTU and the Don't Fragment flag is clear. Each fragment gets its own IPv4 header and carries an identification value plus offset information. The destination reassembles fragments before delivering one UDP datagram to the socket.

Fragment payload sizes except the last must align to eight-byte units. For a 2,000 byte UDP payload over a 1,500 MTU path, the original IPv4 packet is 2,028 bytes. One possible split carries 1,480 bytes of IP payload in the first fragment and 528 bytes in the second. The UDP header appears in the first fragment. User space sees either the reassembled 2,000 byte UDP payload or nothing.

Any missing fragment makes the full datagram unavailable. Fragmentation therefore amplifies loss exposure. Two fragments roughly double the number of network objects whose survival is required for one application message. Fragments also consume reassembly state and may be filtered by firewalls that cannot inspect later fragments as ordinary UDP.

IPv6 routers do not fragment in transit

IPv6 routers do not fragment packets for a sender. If an IPv6 packet is too large for the next hop, a router returns an ICMPv6 Packet Too Big message. The source must reduce its packet size and may use an IPv6 Fragment extension header if the application or stack chooses fragmentation.

This makes path MTU discovery essential. Blocking ICMPv6 Packet Too Big messages can create a black hole where small packets work and larger packets vanish. ICMPv6 is a functional part of IPv6, not optional diagnostic traffic. Firewall policy should permit the required control messages.

IPv6 requires every link to support an MTU of at least 1,280 bytes, although tunnels may handle that requirement through fragmentation at their boundary. A maximum 1,228 byte UDP payload plus 40 bytes of IPv6 and 8 bytes of UDP totals 1,276 bytes. That leaves only four bytes beneath the IPv6 minimum and no room for extension headers. A deployment using IPv6 needs exact on-wire verification rather than a broad assumption based on the word shred.

Path MTU discovery depends on ICMP

For IPv4, a sender can set Don't Fragment and learn from ICMP Destination Unreachable, Fragmentation Needed messages. Linux records route MTU information and returns EMSGSIZE when a datagram exceeds the known path limit on a connected UDP socket. IP_MTU_DISCOVER controls IPv4 path-MTU behavior per socket. IP_MTU can query the current known path MTU for a connected socket.

For IPv6, IPV6_MTU_DISCOVER controls related behavior and IPV6_PATHMTU can return path information. Applications should not assume an asynchronous ICMP error arrives at the same send call that caused it. Connected UDP sockets and the error queue, enabled with IP_RECVERR or IPV6_RECVERR, provide stronger diagnostic information on Linux.

ICMP filtering creates PMTU black holes. Small control packets succeed, while larger datagrams repeatedly fail. tracepath sends probes and reports discovered path-MTU changes without requiring raw-socket privileges on common Linux installations. Its result is evidence for that route and moment, not a permanent path contract.

Linux can cache discovered path information against a route. A corrected network may therefore coexist briefly with stale sender knowledge, while a route change can discard prior knowledge. Record ip route get output during the failure and after recovery. Restarting an application is not a principled PMTU repair, even when creating a new socket changes observed behavior.

Encapsulation consumes the safety margin

VXLAN adds an outer Ethernet, IP, UDP, and VXLAN header. The commonly cited overhead is 50 bytes with outer IPv4 and no extra tags, or 70 bytes with outer IPv6. A physical network supporting only a 1,500 byte IP packet therefore requires an inner MTU near 1,450 for the IPv4 VXLAN case unless the underlay supports larger frames.

WireGuard, IPsec, GRE, Geneve, cloud networking, and service meshes have different overhead and alignment. Encryption can also add variable padding and authentication data. Stacking an overlay inside a VPN compounds overhead. The correct value comes from the deployed path, not one protocol's default.

Inspect ip link show, ip route get 64.130.40.90, tunnel configuration, and cloud interface documentation. If a workload runs in a container, inspect the container interface, bridge, host interface, and underlay. The smallest effective value wins. A host eth0 MTU of 9,000 does not prove a pod veth or remote path supports it.

Fragment reassembly has finite resources

Linux tracks fragments until it can reassemble the original IP packet or a timeout expires. IPv4 thresholds are controlled by net.ipv4.ipfrag_high_thresh and net.ipv4.ipfrag_low_thresh on kernels that expose both, with net.ipv4.ipfrag_time controlling retention time. IPv6 has corresponding net.ipv6.ip6frag_high_thresh, net.ipv6.ip6frag_low_thresh, and net.ipv6.ip6frag_time settings.

Large fragment volumes can consume memory and CPU. When thresholds are exceeded, reassembly entries are pruned or fragments are dropped. nstat exposes counters including IpReasmReqds, IpReasmOKs, IpReasmFails, IpFragOKs, IpFragFails, and IpFragCreates for IPv4. Ip6ReasmReqds, Ip6ReasmOKs, and Ip6ReasmFails cover IPv6 on systems that provide them.

Raising thresholds treats a symptom if a feed should never fragment. The stronger repair is to keep datagrams below the path MTU. Reassembly tuning belongs to workloads that intentionally accept fragmented traffic and have measured the memory and abuse risk.

Packet capture reveals IPv4 fragments

tcpdump can show the IPv4 fragment offset and More Fragments flag. A filter such as 'ip[6:2] & 0x3fff > 0' selects IPv4 fragments. The lower 13 bits contain the offset, and the mask also includes the More Fragments flag. Capturing udp alone can miss later fragments because they do not contain a UDP header at their fragment offset.

Run the fragment filter at both endpoints when possible. A first fragment may decode as UDP, while later fragments appear only as IP fragments. Record tcpdump's dropped-by-kernel summary so capture loss is not mistaken for path loss.

Offloads complicate local views. Generic segmentation offload can show oversized outbound packets before the NIC divides them. Generic receive offload may aggregate work on receive. Capture on an external tap or temporarily test with selected offloads disabled when the exact wire representation matters. Restore production settings after the diagnostic.

The IP MIB records fragmentation behavior

nstat -az can retrieve IP fragmentation and reassembly counters without hand-parsing /proc/net/snmp. IpFragCreates counts fragments created, IpFragOKs counts datagrams fragmented successfully, and IpFragFails counts fragmentation failures. IpReasmReqds counts fragments submitted for reassembly, while IpReasmOKs and IpReasmFails show outcomes.

Counter semantics are host-wide. Other workloads can change them. Take a baseline immediately before a controlled feed test and a second sample afterward. A growing IpReasmReqds on a host that expects only unfragmented shred packets deserves investigation. A growing IpFragCreates on the receiver describes locally transmitted traffic, not incoming feed fragmentation.

The UDP MIB adds context. UdpInErrors and UdpRcvbufErrors identify later receive problems. Clean UDP counters with IpReasmFails rising can still explain missing datagrams because failed reassembly occurs before UDP delivery.

Fragmentation hurts latency variance

A complete unfragmented packet can proceed as soon as it reaches the receive stack. A fragmented datagram waits until every fragment is present. Reordering between fragments adds delay even when nothing is lost. Reassembly also adds table lookup, memory tracking, and copying or buffer management work.

The cost may look small in an average throughput test. The tail expands when one fragment is delayed behind a microburst or processed on a contended CPU. A low-latency receiver has little reason to accept this variance when application packets fit below a conservative MTU.

Larger datagrams can reduce packets per second for bulk transfer, but raw shreds already arrive near their protocol-defined size. Combining them into oversized UDP datagrams would add framing, change failure domains, and force fragmentation on ordinary paths. The packet-rate saving is rarely worth losing independent delivery.

Configure the interface, route, and application together

Changing ip link set dev eth0 mtu 1400 affects packets sent through that interface and the maximum accepted link-layer configuration. It does not rewrite an application's fixed datagram size. A sender using path-MTU discovery may receive EMSGSIZE. A sender allowing IPv4 fragmentation may emit fragments instead.

Route MTU can be set for a specific destination where supported, for example with ip route replace using an mtu attribute. This is an operational override and can go stale after route changes. Fixing the tunnel or underlay MTU is preferable when the path is controlled.

Applications should know their maximum UDP payload, reject unexpected truncation, and expose EMSGSIZE plus error-queue events. Receivers should supply a buffer at least as large as the protocol maximum and check MSG_TRUNC. A successful recvfrom into a short buffer otherwise returns a truncated record that may look like valid partial data.

A conservative shred path stays unfragmented

shredstream.sh publishes a 1,228 byte maximum packet and a 1,216 byte mean packet. Those values sit beneath the ordinary 1,472 byte IPv4 UDP-payload ceiling for a 1,500 MTU path. Even an IPv4 inner MTU of 1,400 leaves 1,372 bytes for UDP payload. The margin narrows under IPv6 and stacked encapsulation.

The operator should verify the destination path rather than enlarge packets because a local LAN supports jumbo frames. Preserve each shred as an independent datagram, permit required ICMP control traffic, and monitor reassembly counters for unexpected fragmentation.

MTU problems are often binary at a size threshold. A health check with a 64 byte datagram can pass while production packets fail. Verification and load tests should include the actual maximum packet size and rate. Reachability at the wrong size is incomplete evidence.