Handle packet bursts

Treat burst rate as a separate workload

The long-window average is 5,585 packets per second at 54.3 Mbps. It does not describe the work presented in one millisecond. Leaders emit groups of data and coding shreds, network queues compress gaps, and a scheduled-out receive thread returns to a full socket. Measure packets per 1, 10, 100, and 1,000 milliseconds. Capacity each stage against the relevant high percentile, not only the one-second mean.

A receiver has three buffers: NIC rings and kernel backlog, the UDP socket queue, and bounded user-space queues. Each protects a different pause. Increasing one does not fix sustained saturation in another.

Drain in batches

Use recvmmsg with 32 or 64 prepared message slots on Linux. One system call can return multiple datagrams and source addresses. Timestamp the batch once when nanosecond precision per packet is unnecessary, or request SO_TIMESTAMPNS ancillary timestamps when kernel arrival time matters. Process all returned messages before waiting again.

Preallocate packet storage. The maximum datagram is 1,228 bytes. A ring of 8,192 slots therefore reserves about 9.6 MiB for raw bytes, plus metadata. Store length, monotonic arrival time, and source address beside each slot. Never allocate a variable-sized vector in the hottest loop unless profiling shows the allocator has enough headroom.

This bounded ring demonstrates the ownership rule without tying it to a networking crate:

use std::collections::VecDeque;

const MAX_PACKET: usize = 1228;

struct Packet { len: usize, bytes: Box<[u8; MAX_PACKET]> }

struct Bounded {
    q: VecDeque<Packet>,
    cap: usize,
    dropped: u64,
}

impl Bounded {
    fn push_newest(&mut self, packet: Packet) {
        if self.q.len() == self.cap {
            self.q.pop_front();
            self.dropped += 1;
        }
        self.q.push_back(packet);
    }
    fn pop(&mut self) -> Option<Packet> { self.q.pop_front() }
}

fn main() {
    let mut ring = Bounded { q: VecDeque::with_capacity(8192), cap: 8192, dropped: 0 };
    ring.push_newest(Packet { len: 0, bytes: Box::new([0; MAX_PACKET]) });
    let packet = ring.pop().unwrap();
    if (packet.len == 0 && packet.bytes[0] == 0) == false { std::process::exit(1); }
}

A concurrent production ring needs synchronization or single-producer and single-consumer ownership. Choose its full policy deliberately. Dropping newest preserves queued state and can help complete older FEC sets. Dropping oldest keeps latency current. A trading system commonly prefers current work, but dropping one shard from many active sets can reduce recovery success. A protocol-aware policy should first discard duplicates, coding shards for already complete sets, expired slots, and work beyond the consumer's latency budget.

Bypass work that is no longer needed

If all data shards for an FEC set arrived, do not wait for coding shards and do not run Reed-Solomon recovery. Once a unique shred ID is accepted, reject later identical payloads before expensive work. Cache verified FEC-level Merkle root and leader signature decisions so members of the same set do not repeat avoidable lookup and verification work, while still validating each proof as required.

Complete serialized data at each DATA_COMPLETE_SHRED boundary instead of waiting for the whole slot. This releases buffers earlier and emits transactions sooner. Preserve slot and index ordering inside each completed range.

Protect the ingest core

Keep logging off the receive thread. A burst of invalid packets must not produce a burst of formatted log messages. Increment reason-labelled counters and sample logs at a bounded rate. Use a metrics exporter that snapshots atomics rather than locking the hot path.

Avoid frequency scaling surprises on dedicated hosts. Record CPU throttling, steal time, run-queue delay, and softirq utilization. Pinning the receive thread can help, but only if the NIC interrupt and other busy tasks are placed intentionally. Test with the production kernel, driver, and virtualization layer.

Increase net.core.netdev_max_backlog only when softnet drops prove the per-CPU backlog is full. Increase NIC ring sizes only when driver counters prove ring exhaustion. Larger early queues can add latency under overload. Every tuning change needs before and after measurements at the same traffic conditions.

Shed stale work

Attach a monotonic arrival time at receive. At every stage, compare item age with a configured maximum. When the queue is overloaded, stop spending CPU on data that the consumer no longer values. Expire whole FEC keys or completed ranges where possible, because scattered packet eviction creates state that can never complete.

Keep correctness separate from freshness. A historical recorder may never discard accepted work and should provision storage and processing for the worst sustained load. A low-latency consumer may discard any item older than 250 milliseconds. Document which mode the receiver implements and expose both loss and age metrics.

Test with controlled pauses

Replay captured datagrams with original timing, then compress timing to two and four times. Inject 50, 100, 250, and 500 millisecond pauses into each stage independently. Verify which counter rises, how quickly queues recover, and whether new work overtakes stale work according to policy.

The pass condition is not only zero process crashes. Require zero kernel drops at the chosen burst target, bounded queue age after the burst, no unbounded memory growth, and a known count for every intentional discard. Repeat after compiler, kernel, NIC, and Agave dependency changes.