Jumbo frames for UDP market data

A jumbo frame is larger than standard Ethernet

Standard Ethernet commonly carries a 1,500 byte IP MTU. A jumbo configuration raises that limit, often to an IP MTU near 9,000 bytes. The exact supported frame size depends on the NIC, switch, provider, encapsulation, and whether documentation counts Ethernet headers or only the IP packet.

The benefit is fewer packets for the same byte volume. Fewer packets mean fewer descriptors, packet headers, interrupt or poll events, stack traversals, and per-packet application operations. Bulk storage, replication, and backup traffic can gain measurable CPU efficiency when both endpoints send large buffers.

Raw Solana shreds do not become larger because an interface supports jumbo frames. shredstream.sh reports a 1,216 byte mean packet and a 1,228 byte maximum. Each shred remains an independent UDP datagram. A 9,000 MTU therefore does not combine seven incoming shreds into one receive event. Any aggregation would require sender, protocol, or host offload behavior beyond the MTU setting.

MTU does not aggregate application datagrams

Changing eth0 from mtu 1500 to mtu 9000 raises the largest IP packet allowed on that interface. It does not cause Linux to concatenate separate UDP datagrams. recvmsg still returns one datagram, and the application still handles 5,585 packets/sec at the measured mean rate.

This point eliminates the strongest claimed benefit for a raw shred receiver. The feed's packet rate drives receive descriptors, UDP lookups, timestamps, and decoder handoffs. A larger receive MTU alone does not reduce that rate. It only permits the host to accept or transmit larger packets for traffic that uses them.

UDP generic receive offload can aggregate processing of compatible datagrams internally on supported kernels and drivers while preserving segmentation metadata. That is an offload feature, not ordinary jumbo-frame behavior. It needs separate validation because capture tools and socket APIs may expose aggregation differently depending on the path.

Header savings matter for large payloads

One full 1,500 byte IPv4 packet with a 20 byte IP header and 8 byte UDP header can carry 1,472 UDP payload bytes. One 9,000 byte packet can carry 8,972 bytes under the same assumptions. Sending 8,972 payload bytes takes seven standard-size UDP datagrams if the application splits them conservatively, but one jumbo datagram when the whole path supports it.

The large datagram saves six sets of headers and six per-packet processing operations. Link-layer efficiency also improves because Ethernet preamble, inter-frame gap, and frame headers occur once. The gain is meaningful at high bulk throughput or very small CPU budgets.

The loss domain becomes larger. One dropped jumbo packet loses nearly 9 KB of application data, and retransmission or recovery has to cover that larger unit. For time-sensitive independent records, combining them also creates head-of-line work at the application layer. Efficiency and fault granularity trade against each other.

Shreds already fit standard paths

A 1,228 byte UDP payload plus 8 bytes of UDP and 20 bytes of IPv4 forms a 1,256 byte IPv4 packet. That leaves 244 bytes under a 1,500 byte path MTU. With IPv6 and no extension headers, the corresponding packet is 1,276 bytes. Standard paths are sufficient under that payload interpretation.

The feed's measured 54.3 Mbps is a modest fraction of a 1 Gbps link, although packet rate and CPU placement still matter. Jumbo frames do not solve socket drops caused by a blocked receive loop or an undersized application queue. They also do not change protocol recovery sets.

A host may use jumbo frames for storage or east-west traffic while accepting ordinary shred packets. That mixed local use is valid if routing and interface design are explicit. The feed need not be padded or repacketized to consume the available MTU.

Every hop must support the larger packet

Jumbo frames work only across an end-to-end domain that supports the selected size. The source NIC, switch ports, trunks, routers, destination NIC, virtual switches, bonds, VLANs, and overlays all matter. One 1,500 MTU segment creates fragmentation, an EMSGSIZE error, or a black hole depending on address family and path-MTU handling.

Cloud providers use different jumbo limits across instance families, availability zones, gateways, and public paths. A virtual machine may show mtu 9001 while an internet gateway supports only 1,500 toward the remote endpoint. The local interface value is not the path MTU.

Operations becomes harder when the domain is partly controlled. A switch may accept a large frame on ingress and drop it on an egress port with a lower maximum frame size. Host counters then look clean. Managed network telemetry or captures on both sides are needed to locate the mismatch.

Encapsulation consumes jumbo capacity too

VXLAN, Geneve, GRE, WireGuard, and IPsec wrap the original packet in outer headers. A 9,000 byte inner packet needs an underlay frame larger than 9,000. Some environments expose an inner MTU such as 8,950 or 8,900 so the outer packet fits the hardware limit. Others provide a 9,000 inner MTU because the physical fabric supports additional headroom.

VLAN tags and stacked tags add link-layer bytes. Hardware described as supporting 9 KB frames may use a maximum that includes or excludes different headers. Consult the exact switch and NIC definitions rather than treating 9,000 and 9,216 as interchangeable.

Container networking adds veth devices, bridges, overlay endpoints, and sometimes encryption. Inspect MTU at every visible layer with ip -d link show. One forgotten veth at 1,500 can defeat a host interface configured for 9,000.

Partial deployment causes asymmetric failures

An MTU mismatch can affect one direction or packet class. Small TCP acknowledgements and control probes work while a large UDP payload fails. TCP may recover by learning a lower maximum segment size through path-MTU discovery, while fixed UDP datagrams continue to exceed the limit. That makes mixed application symptoms plausible.

Asymmetric routing compounds the issue. The forward path may support jumbo frames while the return path for ICMP control messages is filtered. A sender then lacks useful error feedback. For one-way UDP, there is no ordinary data acknowledgement to reveal the fault.

Deploy jumbo frames as a scoped network change, not one host command. Validate both directions, actual application sizes, routed boundaries, failover paths, and monitoring. Record a rollback value. If the path crosses the public internet, assume standard MTU unless the provider offers a specific larger contract.

Validation needs size-controlled probes

IPv4 ping with -M do sets Don't Fragment and can test an exact IP packet size. Under a 9,000 MTU, a ping payload of 8,972 bytes plus 20 bytes of IPv4 and 8 bytes of ICMP forms a 9,000 byte packet. IPv6 uses a 40 byte header, so an 8,952 byte ICMPv6 payload plus 8 bytes of ICMPv6 forms a 9,000 byte IPv6 packet.

ping tests ICMP, not UDP. Policies and routes may differ, but it quickly identifies obvious MTU mismatch. tracepath reports discovered path MTU information. An application-level UDP probe of the production size is stronger for the service path.

Test boundary sizes such as 1,472 and 1,473 byte IPv4 ping payloads on a nominal 1,500 path, then 8,972 on a nominal 9,000 path. A result at only one size turns vague packet loss into a clear threshold. Keep probes within an authorized environment.

Linux configuration is direct and disruptive

ip link set dev eth0 mtu 9000 changes the interface immediately. Active connections and routes can be affected. Network managers, systemd-networkd, Netplan, or cloud initialization may restore a configured value after restart. The persistent configuration belongs in the system's network manager, not an undocumented shell history.

Bonds and VLAN interfaces need compatible MTUs. A child interface generally cannot use a size the parent cannot carry. Bridges and veth pairs also require consistent configuration. ip -d link show lists detailed interface state, while ip route get shows the selected route and can display a route MTU metric when present.

ethtool -S eth0 may expose oversize, length, or giant-frame counters, but names are driver-specific. ip -s link show dev eth0 provides generic RX and TX errors and dropped counts. A successful configuration command does not prove the switch accepts the frames.

CPU benefit must be benchmarked with the real packet mix

Bulk benchmarks that write 64 KB buffers can show a clear jumbo advantage because the sender emits fewer frames. A shred receiver has a fixed small-datagram mix, so the same test does not answer its question. Benchmark the real 1,216 byte mean and 5,585 packets/sec distribution.

Measure per-core cycles, instructions, cache misses, NAPI poll behavior, socket drops, and packet age. perf stat, mpstat -P ALL, /proc/interrupts, and ethtool counters provide complementary views. Keep receive-side scaling, interrupt coalescing, and application affinity identical between runs.

If the jumbo setting changes nothing for the raw feed, that is the expected result. It may still help a separate replication workload on the same network, though shared queues can alter interference. Attribute gains to the traffic that actually emits larger packets.

Offloads provide a different aggregation path

Generic receive offload, large receive offload, generic segmentation offload, TCP segmentation offload, and UDP segmentation offload reduce host processing in different directions and protocols. ethtool -k eth0 shows what the driver supports and which features are active.

TCP large receive offload and segmentation offload do not merge raw UDP messages for the application. UDP GRO and UDP GSO support depends on kernel, driver, and socket configuration. When active, they can reduce per-packet stack work while retaining segment sizing metadata. They can also make packet captures misleading and expose application assumptions about one buffer per datagram.

Treat offload changes separately from MTU changes. Disable features only during a controlled test, record the prior state, and compare correctness as well as throughput. A faster receiver that mishandles GRO segments is not operationally valid.

Jumbo frames can increase latency under contention

A 9,000 byte frame takes six times as long to serialize as a 1,500 byte frame at the same link speed. On 1 Gbps Ethernet, 9,000 bytes alone require about 72 microseconds of serialization, while 1,500 bytes require about 12 microseconds, before link-layer overhead. A small latency-sensitive packet queued behind the large frame waits for that transmission to finish.

At 10 Gbps, the corresponding figures are about 7.2 and 1.2 microseconds. The absolute cost is smaller but can still matter to a tightly budgeted strategy. Priority queues and traffic classes may help, but they introduce another configuration surface and do not preempt a frame already transmitting on ordinary Ethernet.

Jumbo-enabled bulk traffic can therefore worsen the tail for small market-data packets on a shared link. Test mixed traffic, not only isolated throughput. Separate NICs or queues may be more predictable than one interface optimized for incompatible workloads.

Operational simplicity often wins

Standard MTU is widely supported, survives public and private routing, and fits raw shred datagrams. A dedicated receiver gains little from a larger MTU unless other local traffic uses larger application messages or a verified offload path reduces packet work. The cost of one mismatched failover route can exceed the routine CPU saving.

Use jumbo frames inside a controlled fabric when a measured workload benefits and the network team owns every hop. Keep a complete MTU inventory, probe the failover path, monitor giant or length errors, and retain production-size UDP tests.

For shred ingestion, spend effort first on receive queues, NAPI and interrupt placement, socket buffers, batch receive, and decoder handoff. Those mechanisms directly affect the actual packet stream. Jumbo frames are an infrastructure option, not a shred-feed requirement.