Packet loss in a UDP shred receiver

Loss is a location, not one counter

Packet loss means an expected packet did not reach the stage that counts it. That definition changes with the observer. A sender can report every successful send while a switch drops frames. A NIC can receive every frame while the kernel drops packets from a full backlog. The UDP stack can enqueue every datagram while an application discards work from its own full ring.

Start an investigation by drawing the receive path: publisher, source host, network, destination NIC, driver ring, NAPI processing, IP and UDP stack, socket queue, receive loop, decoder queue, and protocol state. Place a counter or packet capture at as many boundaries as practical. The goal is to find the first stage where expected sequence identities disappear.

No single Linux counter proves end-to-end loss. Protocol sequence gaps are closest to the application truth, while host counters locate local failure. Correlating both is the central method. Tuning before localization often moves the drop to another queue and makes the incident harder to explain.

Sequence gaps show pipeline completeness

UDP has no transport sequence number. The application protocol must provide identity. Solana shreds include slot and index information; source packets and parity packets serve distinct recovery roles. A receiver records which identities arrive, accepts bounded reordering, removes duplicates, and determines which expected items remain absent after its waiting rule.

Arrival order alone is insufficient. Packet 102 followed by packet 104 is not immediately proof that 103 is lost because 103 may be reordered. Declaring loss too early inflates gaps. Waiting too long delays decoding. Measure the actual reorder-depth and delay distribution, then choose a bound that fits the strategy and recovery process.

Count raw missing observations separately from unrecoverable protocol data. Coding shreds can rebuild some missing data shreds. A raw packet-loss rate can therefore exceed the block-data loss rate. Recovery consumes CPU and time, so it is not free. Report received, duplicate, reordered, reconstructed, and unrecoverable counts independently.

Sender loss can happen before the wire

A UDP publisher can drop packets in user space, its socket send buffer, qdisc, driver, or NIC. A successful send call commonly means local acceptance, not transmission. Under local pressure, a nonblocking socket may report EAGAIN, and the networking stack can report ENOBUFS in some conditions. Applications must count attempted, accepted, failed, and transmitted work where hardware counters permit.

The source qdisc can drop when its queue limit is reached. tc -s qdisc show dev eth0 reports packets, bytes, drops, overlimits, and backlog for configured queue disciplines. ethtool -S eth0 provides driver-specific transmit counters. Packet capture on the source host narrows whether a datagram reached the capture hook, but capture placement and offloads affect what is visible.

For a fanout service, per-destination success matters. One bad route or neighbor entry can affect one subscriber without affecting others. Aggregate send-rate telemetry can hide that pattern. A provider needs destination-level error and pacing statistics even though UDP offers no delivery acknowledgement.

The network can discard or reorder

Routers and switches drop when queues overflow, policies reject traffic, checksums fail, or packets exceed an MTU under incompatible fragmentation rules. Equal-cost multipath changes, link errors, and microbursts can also produce reordering or loss. Public internet paths expose more uncontrolled queues than a cross-connect or private fabric.

Two packet captures, one near each endpoint, are the strongest direct localization method when clocks and packet identities can be aligned. A packet present at the source capture and absent at the destination capture was lost somewhere between those observation points. Captures can also drop, so inspect every tcpdump exit count: captured packets, filter receipts, and kernel drops.

ping is a weak proxy for UDP feed health. ICMP may follow different policies, uses a different packet size and rate, and can be deprioritized. A clean ping does not prove a sustained UDP stream is clean. An impaired ping is still useful evidence of broad path trouble.

NIC rings are the first destination queue

The NIC places received data into buffers described by a receive ring. If descriptors or buffers are unavailable, the NIC or driver drops before the normal IP stack. Where implemented, ethtool -g eth0 reports current ring sizes and supported maxima. On supported hardware, change the receive ring with:

ethtool -G eth0 rx 4096

Counter names vary. Intel, Mellanox, ENA, virtio, and other drivers expose different fields through ethtool -S. Look for missed receives, unavailable buffers, discards, generic drops, and queue-specific variants. Never build a portable alert by assuming one exact vendor field. Inventory the deployed interface and verify a counter with a controlled overload.

A larger ring absorbs longer service gaps but can increase queueing and cache footprint. If the NAPI poller cannot sustain the offered packet rate, the ring eventually fills at any finite size. Ring tuning is burst accommodation, not a throughput substitute.

Softnet drops indicate stack pressure

Linux processes received packets through NAPI and per-CPU networking structures. /proc/net/softnet_stat supplies one hexadecimal row for each CPU. Field two counts full-backlog drops; field three counts polling cycles that exhaust their processing budget. Later fields depend on kernel version.

Convert counter deltas, not the total alone. A host may have accumulated softnet drops months before the incident. Bracket each test with samples, map rows to CPUs, and correlate them with interrupt and receive-queue placement. One hot CPU can drop while total host CPU utilization remains low.

The input backlog's packet limit uses this sysctl:

net.core.netdev_max_backlog

Packet work per polling cycle is bounded by net.core.netdev_budget. On relevant paths, a separate time limit comes from:

net.core.netdev_budget_usecs

Raising them can reduce backlog pressure but can also let networking monopolize a CPU longer. Scheduler latency for the application must be measured after changes.

UDP receive errors locate socket pressure

System-wide UDP MIB counters are available through nstat or /proc/net/snmp. UdpInDatagrams counts successful delivery to user space, while UdpInErrors covers receive errors. UdpRcvbufErrors identifies datagrams dropped because a UDP socket receive buffer was full. UdpNoPorts counts datagrams addressed to a port with no listener.

The final field in each /proc/net/udp socket row is drops. Match the local address and hexadecimal port to isolate a listener. Detailed socket memory comes from ss, whose skmem tuple can show receive allocation, limit, and a supported drop field.

SO_RCVBUF sets the socket request, subject to the unprivileged ceiling in net.core.rmem_max. Reported values are doubled by Linux for bookkeeping. A larger buffer helps only if the application later catches up. If UdpRcvbufErrors rises continuously during steady load, improve drain throughput before allocating a deeper delay reservoir.

Application loss is still packet loss to the strategy

A clean kernel handoff does not mean the event reached decoding. The receive loop may reject an unexpected length, fail a checksum or signature rule, find a full user-space ring, or discard an old packet. Workers may crash, time out, or invalidate an entire recovery set after partial processing.

Every discard branch needs a named counter. Useful names include rx_datagrams, rx_truncated, rx_bad_length, rx_enqueue_full, rx_duplicate, rx_reordered, fec_recovered, fec_unrecoverable, decode_failed, and stale_discarded. Labels should stay bounded. Do not attach slot or signature as a metrics label.

Log samples can preserve identities for diagnosis, but per-packet logging is dangerous at sustained feed rates. A synchronous formatter or disk write can create the loss it is trying to explain. Use counters, histograms, and a bounded diagnostic sample buffer.

Fragmentation creates correlated loss

An IP datagram larger than the path MTU may be fragmented in IPv4 or rejected so path MTU discovery can reduce the size. With IPv6, transit routers never perform fragmentation. If a fragmented UDP datagram loses any one fragment, the entire UDP datagram is unavailable to the application. Fragments also use reassembly memory and timeouts on the destination.

Raw shred packets are designed for ordinary Ethernet paths, and shredstream.sh reports 1,228 bytes as the packet maximum. If that size denotes UDP payload, adding 20 IPv4 bytes and 8 UDP bytes still stays below an IP MTU of 1,500 bytes. Operators should confirm capture semantics and encapsulation rather than infer every layer from one size number.

Tunnels reduce effective MTU. VXLAN, WireGuard, IPsec, GRE, and cloud overlays add headers. A packet safe on bare Ethernet can fragment or fail through an overlay with a smaller inner MTU. ip route get and tracepath help inspect route and path-MTU information, while tcpdump can reveal IPv4 fragments.

Offloads can confuse observation

Receive aggregation, including UDP GRO, can combine packet processing inside the host. Transmit segmentation offload can make outbound captures show packets larger than those placed on the wire. Checksum offload can cause tcpdump on the transmitting host to label a checksum incorrect because hardware has not filled it yet.

These effects do not inherently mean data loss. They change where packet boundaries and completed headers are visible. ethtool -k eth0 lists offload features such as generic-receive-offload, generic-segmentation-offload, rx-checksumming, and receive-hashing. Disable one feature only for a controlled diagnostic if evidence requires it.

Capture drops are another source of false conclusions. tcpdump reports packets dropped by kernel on exit. Increase its capture buffer with -B for a test, restrict the filter, and avoid decoding or writing more data than needed. A lossy observer cannot establish exact feed loss.

Loss percentage needs a denominator and window

A statement such as 0.01 percent loss is incomplete without the expected count, interval, treatment of duplicates, and treatment of recoveries. Consider a stream tested at 6,000 packets/sec. Its one-minute expected count is 360,000, so 0.01 percent raw loss represents 36 missing packets. Distribution matters more than the percentage alone.

Thirty-six isolated missing coding shreds may be fully recoverable. Thirty-six adjacent data shreds in one recovery set may not be. Even a minute-long average can hide a 20 ms burst that invalidates one critical slot. Report burst length, maximum consecutive gap, affected slots, and recovery outcome.

Use confidence intervals for low loss rates and tests long enough to observe rare events. A ten-second clean run gives little evidence about a service expected to run for a month. Long tests also expose route changes, housekeeping, log rotation, and thermal behavior absent from a short benchmark.

Controlled fault injection validates the monitors

Linux netem can introduce loss, delay, duplication, corruption, and reordering on a test interface. Use a dedicated host or isolated namespace because changing the root qdisc affects matching traffic on that interface. Record the existing qdisc and remove the test rule afterward.

An imposed 0.1 percent random loss should appear as application sequence gaps. It should not increase destination UdpRcvbufErrors if the host keeps up, because netem drops before the datagrams reach that socket. A forced receiver pause should show the opposite pattern once the socket queue fills. These two tests validate localization.

Random independent loss does not model every production incident. netem supports correlation and burst models, but hardware microbursts, reroutes, and CPU stalls have different shapes. Build several scenarios and retain the counter deltas with each result.

The repair follows the first exhausted stage

Sender qdisc drops call for source pacing, queue policy, or more transmit capacity. Network-only loss calls for path investigation, provider escalation, or diverse routes. NIC misses call for ring, interrupt, CPU, and driver analysis. Softnet drops call for per-CPU receive-path work. UdpRcvbufErrors call for socket drain and buffer analysis. User-ring drops call for pipeline capacity or shedding policy.

Changing every sysctl at once destroys causal evidence. Capture a baseline, change one constraint, replay the same workload, and compare loss plus latency. A change that removes drops but adds unacceptable residence time is not a complete fix.

The final outcome belongs at the protocol layer. The receiver must know whether missing shreds were reconstructed, whether the slot remained decodable, and whether dependent trading state was invalidated. Infrastructure counters explain the mechanism. They do not decide semantic safety.