Detect dropped UDP shreds

Measure every boundary

A missing shred is an observation, not a diagnosis. UDP provides no sender acknowledgement, sequence stream, or retransmission. Compare counters at the NIC, kernel backlog, UDP stack, socket, receive loop, and shred parser. Take snapshots over the same interval and calculate deltas. Cumulative values from different boot times cannot be compared directly.

The feed averages 5,585 packets per second. A one-minute interval therefore carries roughly 335,100 packets, but block production varies and this estimate is not an integrity check. Use shred indices and FEC-set membership for protocol-level gaps.

Read the UDP MIB counters

Linux exports named UDP counters through /proc/net/snmp. The exact relevant names are Udp:InDatagrams, Udp:NoPorts, Udp:InErrors, Udp:RcvbufErrors, Udp:InCsumErrors, Udp:IgnoredMulti, and, on kernels that expose it, Udp:MemErrors.

RcvbufErrors counts datagrams discarded because a UDP receive queue had no space. InCsumErrors counts checksum failures. NoPorts counts arrivals for an unbound UDP destination. InErrors is the broader UDP input error count and includes receive-buffer and checksum errors on Linux, so do not sum it with its components as though they were independent losses.

Read named fields instead of column numbers:

use std::{collections::BTreeMap, fs, io};

fn udp_snmp() -> io::Result<BTreeMap<String, u64>> {
    let text = fs::read_to_string("/proc/net/snmp")?;
    let mut lines = text.lines();
    while let Some(header) = lines.next() {
        let Some(values) = lines.next() else { break };
        if header.starts_with("Udp:") && values.starts_with("Udp:") {
            let names = header.split_whitespace().skip(1);
            let nums = values.split_whitespace().skip(1);
            return names.zip(nums).map(|(name, value)| {
                value.parse::<u64>()
                    .map(|n| (name.to_owned(), n))
                    .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))
            }).collect();
        }
    }
    Err(io::Error::new(io::ErrorKind::NotFound, "Udp section missing"))
}

fn main() -> io::Result<()> {
    let counters = udp_snmp()?;
    let _receive_errors = counters.get("RcvbufErrors").copied().unwrap_or(0);
    Ok(())
}

The nstat command reads the same family of kernel statistics with stable names:

nstat -az UdpInDatagrams UdpInErrors UdpRcvbufErrors UdpInCsumErrors UdpNoPorts

Resetting nstat's local history does not reset kernel counters. A monitoring agent should store the prior sample and handle host reboot or counter reset.

Read per-socket overflow

Enable SO_RXQ_OVFL before binding. Linux then attaches a SOL_SOCKET, SO_RXQ_OVFL control message to received datagrams after an overflow. Its value is a cumulative 32-bit drop count for that socket. Use unsigned wrapping subtraction between samples. This is the best attribution for queue overflow on the feed socket.

Inspect the same socket externally with:

ss -u -l -n -m -p 'sport = :8001'

In the skmem tuple, r is current receive memory, rb is the receive-buffer limit, and d is socket drops. The final field of the corresponding line in /proc/net/udp is also the socket drop count, but mapping an inode and decoding columns yourself is more fragile than ss or ancillary data.

Check softnet and NIC loss

/proc/net/softnet_stat is per CPU and hexadecimal. In the traditional layout, field 1 is processed, field 2 is dropped because the per-CPU input backlog was full, and field 3 is time_squeeze, where the networking poll exhausted its work or time budget. Kernel versions can add fields, so use a maintained collector that labels the format for the running kernel. Rising second-field drops point before UDP socket delivery.

Use ip -s -s link show dev eth0 for standard link counters. Watch RX dropped, errors, missed, overrun, and crc when available. Use ethtool -S eth0 for driver queue counters. Names such as rx_missed_errors, rx_no_buffer_count, and per-queue rx_queue_N_drops are driver-specific. Discover and record the actual names on the deployed NIC rather than claiming one driver name is universal.

Count inside the application

Increment receiver_datagrams_total immediately after a successful receive, before parsing. Increment separate counters for wrong source, truncated datagram, invalid shred, signature failure, duplicate, queue-full discard, FEC-recovered data shred, and expired incomplete set. Keep bytes_received_total based on the returned datagram length.

Protocol gaps need two levels. A missing data index within a slot is provisional because UDP reorders packets and coding shards may recover it. Count data_index_gap_observed_total when a gap first appears, then data_shred_recovered_total when recovery fills it, and data_shred_unrecoverable_total only when the set expires without enough unique shards.

Do not infer source loss from data-index gaps alone. Coding-shred indices form a separate sequence, packets can be duplicated, slots can end with a short FEC set, and first observation may begin mid-slot after a receiver restart.

Attribute by counter pattern

NIC RX missed or no-buffer increases mean the host did not deliver frames into the network stack. Softnet drops mean CPU backlog pressure after the NIC. UdpInCsumErrors means corrupted packets were rejected. UdpRcvbufErrors and per-socket overflow mean the application did not drain the queue fast enough. Application queue drops mean the socket was drained but the next stage was saturated.

If none of these rise while final unrecoverable gaps increase, loss occurred before the destination host, the receiver started late, or invalid packets were rejected by protocol checks. Compare a short packet capture with application counts. Capture only long enough to diagnose because packet capture itself consumes CPU and memory bandwidth.