Packet loss in a UDP shred receiver

Loss is a location, not one counter

Packet loss means an expected packet did not reach the stage that counts it. That definition changes with the observer. A sender can report every successful send while a switch drops frames. A NIC can receive every frame while the kernel drops packets from a full backlog. The UDP stack can enqueue every datagram while an application discards work from its own full ring.

Start an investigation by drawing the receive path: publisher, source host, network, destination NIC, driver ring, NAPI processing, IP and UDP stack, socket queue, receive loop, decoder queue, and protocol state. Place a counter or packet capture at as many boundaries as practical. The goal is to find the first stage where expected sequence identities disappear.

No single Linux counter proves end-to-end loss. Protocol sequence gaps are closest to the application truth, while host counters locate local failure. Correlating both is the central method. Tuning before localization often moves the drop to another queue and makes the incident harder to explain.

Sequence gaps are the end-to-end signal

UDP has no transport sequence number. The application protocol must provide identity. Solana shreds include slot and index information, and data and coding shreds have distinct roles within recovery sets. A receiver records which identities arrive, accepts bounded reordering, removes duplicates, and determines which expected items remain absent after its waiting rule.

Arrival order alone is insufficient. Packet 102 followed by packet 104 is not immediately proof that 103 is lost because 103 may be reordered. Declaring loss too early inflates gaps. Waiting too long delays decoding. Measure the actual reorder-depth and delay distribution, then choose a bound that fits the strategy and recovery process.

Count raw missing observations separately from unrecoverable protocol data. Coding shreds can rebuild some missing data shreds. A raw packet-loss rate can therefore exceed the block-data loss rate. Recovery consumes CPU and time, so it is not free. Report received, duplicate, reordered, reconstructed, and unrecoverable counts independently.

Sender loss can happen before the wire

A UDP publisher can drop packets in user space, its socket send buffer, qdisc, driver, or NIC. A successful send call commonly means local acceptance, not transmission. Under local pressure, a nonblocking socket may report EAGAIN, and the networking stack can report ENOBUFS in some conditions. Applications must count attempted, accepted, failed, and transmitted work where hardware counters permit.

The source qdisc can drop when its queue limit is reached. tc -s qdisc show dev eth0 reports packets, bytes, drops, overlimits, and backlog for configured queue disciplines. ethtool -S eth0 provides driver-specific transmit counters. Packet capture on the source host narrows whether a datagram reached the capture hook, but capture placement and offloads affect what is visible.

For a fanout service, per-destination success matters. One bad route or neighbor entry can affect one subscriber without affecting others. Aggregate send-rate telemetry can hide that pattern. A provider needs destination-level error and pacing statistics even though UDP offers no delivery acknowledgement.

The network can discard or reorder

Routers and switches drop when queues overflow, policies reject traffic, checksums fail, or packets exceed an MTU under incompatible fragmentation rules. Equal-cost multipath changes, link errors, and microbursts can also produce reordering or loss. Public internet paths expose more uncontrolled queues than a cross-connect or private fabric.

Two packet captures, one near each endpoint, are the strongest direct localization method when clocks and packet identities can be aligned. A packet present at the source capture and absent at the destination capture was lost somewhere between those observation points. Captures themselves can drop, so inspect tcpdump's captured, received by filter, and dropped by kernel summary.

ping is a weak proxy for UDP feed health. ICMP may follow different policies, uses a different packet size and rate, and can be deprioritized. A clean ping does not prove a 5,585 packets/sec stream is clean. An impaired ping is still useful evidence of broad path trouble.

NIC rings are the first destination queue

The NIC places received data into buffers described by a receive ring. If descriptors or buffers are unavailable, the NIC or driver drops before the normal IP stack. ethtool -g eth0 shows supported and current ring sizes where the driver implements the operation. ethtool -G eth0 rx 4096 changes the receive ring on supported hardware.

Counter names vary. Intel, Mellanox, ENA, virtio, and other drivers expose different fields through ethtool -S. Common patterns include rx_missed_errors, rx_no_buffer_count, rx_discards, rx_dropped, and per-queue variants. Never build a portable alert by assuming one exact vendor field. Inventory the deployed interface and verify a counter with a controlled overload.

A larger ring absorbs longer service gaps but can increase queueing and cache footprint. If the NAPI poller cannot sustain the offered packet rate, the ring eventually fills at any finite size. Ring tuning is burst accommodation, not a throughput substitute.

Softnet drops indicate stack pressure

Linux processes received packets through NAPI and per-CPU networking structures. /proc/net/softnet_stat contains one row per CPU in hexadecimal. The second field counts drops associated with a full input backlog. The third field counts times the processing budget was exhausted. Later fields depend on kernel version.

Convert counter deltas, not the total alone. A host may have accumulated softnet drops months before the incident. Sample before and after a test, map rows to CPUs, and correlate with interrupt and receive-queue placement. One hot CPU can drop while total host CPU utilization remains low.

net.core.netdev_max_backlog controls the maximum packet count in the input backlog. net.core.netdev_budget and net.core.netdev_budget_usecs bound work per network polling cycle on relevant paths. Raising them can reduce backlog pressure but can also let networking monopolize a CPU longer. Scheduler latency for the application must be measured after changes.

UDP receive errors locate socket pressure

The UDP stack reports system-wide counters through /proc/net/snmp and tools such as nstat. UdpInDatagrams counts datagrams delivered to users, while UdpInErrors covers receive errors. UdpRcvbufErrors identifies datagrams dropped because a UDP socket receive buffer was full. UdpNoPorts counts datagrams addressed to a port with no listener.

The final field in each /proc/net/udp socket row is drops. Match the local address and hexadecimal port to isolate a listener. ss -u -a -n -m can print socket memory state, with a skmem tuple containing current receive allocation, receive limit, and a drop field where supported.

SO_RCVBUF sets the socket request, and net.core.rmem_max caps unprivileged requests. Linux doubles the requested value for bookkeeping when reporting it. A larger buffer helps only if the application later catches up. If UdpRcvbufErrors rises continuously during steady load, improve drain throughput before allocating a deeper delay reservoir.

Application loss is still packet loss to the strategy

A clean kernel handoff does not mean the event reached decoding. The receive loop may reject an unexpected length, fail a checksum or signature rule, find a full user-space ring, or discard an old packet. Workers may crash, time out, or invalidate an entire recovery set after partial processing.

Every discard branch needs a named counter. Useful names include rx_datagrams, rx_truncated, rx_bad_length, rx_enqueue_full, rx_duplicate, rx_reordered, fec_recovered, fec_unrecoverable, decode_failed, and stale_discarded. Labels should stay bounded. Do not attach slot or signature as a metrics label.

Log samples can preserve identities for diagnosis, but per-packet logging is dangerous at 5,585 packets/sec. A synchronous formatter or disk write can create the loss it is trying to explain. Use counters, histograms, and a bounded diagnostic sample buffer.

Fragmentation creates correlated loss

An IP datagram larger than the path MTU may be fragmented in IPv4 or rejected so path MTU discovery can reduce the size. IPv6 routers do not fragment in transit. If a fragmented UDP datagram loses any one fragment, the entire UDP datagram is unavailable to the application. Fragments also use reassembly memory and timeouts on the destination.

Raw shred packets are designed to fit ordinary Ethernet paths, and shredstream.sh reports a maximum packet size of 1,228 bytes. With a 20 byte IPv4 header and 8 byte UDP header, that remains below a 1,500 byte IP MTU if the product size refers to UDP payload. Operators should confirm capture semantics and encapsulation rather than infer every layer from one size number.

Tunnels reduce effective MTU. VXLAN, WireGuard, IPsec, GRE, and cloud overlays add headers. A packet safe on bare Ethernet can fragment or fail through an overlay with a smaller inner MTU. ip route get and tracepath help inspect route and path-MTU information, while tcpdump can reveal IPv4 fragments.

Offloads can confuse observation

Generic receive offload and UDP GRO can combine packet processing inside the host. Generic segmentation offload can make outbound captures show packets larger than those placed on the wire. Checksum offload can cause tcpdump on the transmitting host to label a checksum incorrect because hardware has not filled it yet.

These effects do not inherently mean data loss. They change where packet boundaries and completed headers are visible. ethtool -k eth0 lists offload features such as generic-receive-offload, generic-segmentation-offload, rx-checksumming, and receive-hashing. Disable one feature only for a controlled diagnostic if evidence requires it.

Capture drops are another source of false conclusions. tcpdump reports packets dropped by kernel on exit. Increase its capture buffer with -B for a test, restrict the filter, and avoid decoding or writing more data than needed. A lossy observer cannot establish exact feed loss.

Loss percentage needs a denominator and window

A statement such as 0.01 percent loss is incomplete without the expected count, interval, treatment of duplicates, and treatment of recoveries. At 5,585 packets/sec, one minute contains about 335,100 packets. A 0.01 percent raw loss rate represents about 34 missing packets in that minute. Distribution matters more than the percentage alone.

Thirty-four isolated missing coding shreds may be fully recoverable. Thirty-four adjacent data shreds in one recovery set may not be. A one-minute average can also hide a 20 ms burst that invalidates one critical slot. Report burst length, maximum consecutive gap, affected slots, and recovery outcome.

Use confidence intervals for low loss rates and tests long enough to observe rare events. A ten-second clean run gives little evidence about a service expected to run for a month. Long tests also expose route changes, housekeeping, log rotation, and thermal behavior absent from a short benchmark.

Controlled fault injection validates the monitors

Linux netem can introduce loss, delay, duplication, corruption, and reordering on a test interface. Use a dedicated host or isolated namespace because changing the root qdisc affects matching traffic on that interface. Record the existing qdisc and remove the test rule afterward.

An imposed 0.1 percent random loss should appear as application sequence gaps. It should not increase destination UdpRcvbufErrors if the host keeps up, because netem drops before the datagrams reach that socket. A forced receiver pause should show the opposite pattern once the socket queue fills. These two tests validate localization.

Random independent loss does not model every production incident. netem supports correlation and burst models, but hardware microbursts, reroutes, and CPU stalls have different shapes. Build several scenarios and retain the counter deltas with each result.

The repair follows the first exhausted stage

Sender qdisc drops call for source pacing, queue policy, or more transmit capacity. Network-only loss calls for path investigation, provider escalation, or diverse routes. NIC misses call for ring, interrupt, CPU, and driver analysis. Softnet drops call for per-CPU receive-path work. UdpRcvbufErrors call for socket drain and buffer analysis. User-ring drops call for pipeline capacity or shedding policy.

Changing every sysctl at once destroys causal evidence. Capture a baseline, change one constraint, replay the same workload, and compare loss plus latency. A change that removes drops but adds unacceptable residence time is not a complete fix.

The final outcome belongs at the protocol layer. The receiver must know whether missing shreds were reconstructed, whether the slot remained decodable, and whether dependent trading state was invalidated. Infrastructure counters explain the mechanism. They do not decide semantic safety.