Size the UDP receive buffer

Start with the measured packet rate

The feed averages 5,585 packets per second and 1,216 bytes per packet. A receive queue must cover the period during which the application cannot drain the socket. Call that period the stall budget. Include scheduler delay, stop-the-world pauses in linked runtimes, short CPU contention, and the time workers spend catching up after a burst.

For 250 milliseconds at the average rate:

packets = 5,585 packets/s × 0.250 s = 1,396.25 packets
payload = 1,397 packets × 1,216 bytes = 1,698,752 bytes
overhead allowance = 1,397 packets × 512 bytes = 715,264 bytes
burst multiplier = 4
required queue = (1,698,752 + 715,264) × 4 = 9,656,064 bytes
round up = 16 MiB requested SO_RCVBUF

The 512-byte allowance is a conservative planning value for socket-buffer accounting, packet metadata, alignment, and variance. It is not a Solana wire constant. Linux charges receive memory by internal allocation size rather than UDP payload length alone, and the exact charge depends on kernel, architecture, and driver path. Validate it with ss -u -m and live traffic.

A request of 16 MiB covers roughly 2.4 seconds of mean payload bytes if no overhead is counted, or about 1.7 seconds using 1,728 charged bytes per packet. After applying the four-times burst assumption, it represents about 430 milliseconds at that stressed packet rate. This exceeds the original 250 millisecond stall target with rounding headroom.

Set both the host ceiling and socket request

Linux limits an ordinary SO_RCVBUF request with net.core.rmem_max. Set the ceiling above the requested value. Linux doubles the requested value internally and returns that doubled result from getsockopt. For a 16 MiB request, use a ceiling of at least 16 MiB and expect a readback near 32 MiB.

sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.rmem_default=4194304
sysctl net.core.rmem_max net.core.rmem_default

Persist the settings through the host's sysctl configuration system. rmem_default affects sockets that do not set their own buffer. The receiver should always set SO_RCVBUF, so rmem_max is the important ceiling. Avoid making the global default enormous because every UDP socket can consume receive memory.

Use socket2 and verify the effective value:

use socket2::{Domain, Protocol, Socket, Type};
use std::io;

fn make_socket(requested: usize) -> io::Result<Socket> {
    let sock = Socket::new(Domain::IPV4, Type::DGRAM, Some(Protocol::UDP))?;
    sock.set_recv_buffer_size(requested)?;
    let effective = sock.recv_buffer_size()?;
    if effective < requested {
        return Err(io::Error::other("receive buffer capped below request"));
    }
    let _verified_readback = effective;
    Ok(sock)
}

fn main() -> io::Result<()> {
    let _socket = make_socket(16 * 1024 * 1024)?;
    Ok(())
}

Because Linux usually doubles the value, checking effective < requested detects a severe cap but does not prove exact success. Record both values and alert when the readback changes across deployments.

Separate kernel and application buffers

SO_RCVBUF protects only the interval before the receive system call copies a datagram into user space. Once received, the datagram occupies an application object or queue. Size that queue independently in packets, not bytes alone.

At four times the measured rate, 250 milliseconds is:

5,585 × 4 × 0.250 = 5,585 packet slots
5,585 × 1,228 maximum bytes = 6,859,380 bytes of packet storage

Round to 8,192 slots if the queue implementation benefits from a power of two. Include object metadata and allocator overhead in the memory budget. A bounded queue gives a precise overload signal. An unbounded queue trades visible packet loss for growing latency and eventual memory failure.

Do not add kernel and application capacity and claim the sum as one clean stall budget. The receive thread must run to move packets from the kernel into the application queue. If that thread is descheduled, only SO_RCVBUF helps. If workers are slow while the receive thread runs, the application queue helps.

Measure the real burst distribution

Count datagrams in 1 millisecond, 10 millisecond, 100 millisecond, and 1 second windows. Retain the maximum and high percentiles. Feed production is not uniform. Coding shreds and data shreds can arrive close together, and network queues can compress inter-arrival gaps.

Increase the multiplier when the measured maximum exceeds four times the long-window mean. Decrease it only after observing busy periods and verifying zero RcvbufErrors. A large receive buffer does not repair a receiver that is continuously slower than the feed. It delays loss and increases the amount of stale data waiting in the kernel.

Bound latency as well as loss

Timestamp immediately after receive. Export the age of the oldest application-queued datagram. If that age exceeds the strategy's usefulness window, drop stale work intentionally and continue at the head. For raw shred consumers, processing seconds-old data while current packets are being discarded is usually the wrong failure mode.

Run a stall injection test. Pause the receive thread for 50, 100, 250, and 500 milliseconds under representative traffic. Observe SO_RXQ_OVFL, Udp:RcvbufErrors, and application sequence gaps. The configured buffer is accepted only when it covers the stated target on the deployed kernel and host.