Linux kernel receive buffers for UDP

The receive buffer is not one buffer

Operators often say receive buffer while referring to several queues. The NIC has receive descriptors and data buffers. Linux can queue packets in a per-CPU input backlog. The UDP socket has memory accounting and a receive queue. The application frequently adds another ring between ingest and decoding. Each stage has a different limit and counter.

Increasing net.core.rmem_max affects the ceiling for a socket's receive-buffer request. It does not enlarge the NIC ring, netdev backlog, or user-space queue. Increasing net.core.netdev_max_backlog affects a queue before UDP socket lookup. It does not give a slow application unlimited time.

Tune by finding the first queue that fills. A NIC counter that rises calls for driver, ring, NAPI, or CPU work. A softnet drop calls for stack processing analysis. UdpRcvbufErrors calls for socket drain analysis. A clean kernel with application ring rejects calls for user-space work. One large sysctl bundle hides that distinction.

SO_RCVBUF is the socket request

An application calls setsockopt with SOL_SOCKET and SO_RCVBUF to request receive memory for a socket. Linux doubles the requested value for internal bookkeeping and returns that doubled figure through getsockopt. ss -m also shows the doubled receive limit in its skmem output. This behavior regularly causes operators to think a setting was applied twice.

If a process requests 4,194,304 bytes, a successful inspection can report 8,388,608. The request is limited by net.core.rmem_max for ordinary processes. An application with the required privilege can use SO_RCVBUFFORCE to exceed that ceiling, but system-wide sizing should be intentional rather than delegated to an unexpected binary.

Set the buffer in application initialization and fail or warn when the effective value is below the required minimum. Relying on a shell sysctl alone is incomplete because it changes the allowed ceiling, not the socket's requested allocation.

TCP receive autotuning does not size a UDP listener from observed bandwidth-delay product. Parameters under net.ipv4.tcp_rmem belong to TCP and do not replace SO_RCVBUF. A UDP application that never requests memory receives the configured default, regardless of feed rate. This distinction should be asserted in startup telemetry so a packaging regression cannot silently return the socket to a small default.

rmem_default and rmem_max have different roles

net.core.rmem_default supplies the default receive-buffer value for sockets that do not set SO_RCVBUF. net.core.rmem_max caps normal SO_RCVBUF requests. Setting rmem_max to 134217728 does not make every UDP socket allocate or reserve 128 MiB. Memory is accounted as packets arrive, subject to the configured limit.

Raising rmem_default globally affects unrelated sockets and can increase the amount of memory available to workloads that never asked for it. A dedicated receiver should request its own known value. Keep the default conservative unless host-wide workloads have a measured need.

Read active values with sysctl net.core.rmem_default net.core.rmem_max. Persist approved settings through the host's sysctl configuration management. A command-line sysctl -w is temporary and can disappear after restart or image replacement.

UDP also has protocol memory pressure controls

Linux exposes net.ipv4.udp_mem as three values measured in memory pages: pressure begins above the first threshold, the second is a pressure target, and the third is the maximum number of pages queued by all UDP sockets. Kernel behavior and automatic defaults vary with system memory and version.

net.ipv4.udp_rmem_min specifies the minimum receive-buffer size each UDP socket can use while UDP is under memory pressure. net.ipv4.udp_wmem_min provides the corresponding send-side minimum. These protocol-wide controls are not substitutes for SO_RCVBUF.

Inspect values before changing them:

sysctl net.ipv4.udp_mem sysctl net.ipv4.udp_rmem_min net.ipv4.udp_wmem_min getconf PAGESIZE

A udp_mem value cannot be interpreted as bytes until multiplied by the host page size. Copying a three-number value from a different machine ignores available RAM, page size, kernel defaults, and every other UDP workload.

The socket queue holds charged memory

Socket receive accounting includes more than UDP payload. Packet metadata, alignment, allocator behavior, and skb overhead consume space. Dividing SO_RCVBUF by the mean 1,216 byte packet size overestimates how many packets fit. The error grows for smaller datagrams because fixed per-packet overhead is a larger share.

Measure the real relationship. Pause the receiver for a known interval under a known packet distribution, observe queued memory with ss, then resume before loss if the test is safe. Repeat until the buffer's time coverage is understood. Kernel, architecture, offload, and packet layout changes can alter the result.

Buffering is a time budget. At 5,585 packets/sec, a 10 ms pause exposes about 56 packets, 100 ms exposes about 559, and one second exposes about 5,585. The chosen allocation should cover defined transient pauses with margin, not an indefinite service-rate deficit.

ss reports current socket memory

ss -u -a -n -m prints UDP sockets and memory details. A skmem tuple commonly contains r for receive memory allocated, rb for receive-buffer limit, t and tb for transmit allocation and limit, f for forward-allocated memory, w for queued write memory, o for option memory, bl for backlog memory, and d for drops.

Fields depend on kernel and socket state. A line such as skmem:(r1216000,rb8388608,t0,tb212992,f2304,w0,o0,bl0,d17) shows receive allocation near 1.216 MB, an 8 MiB reported limit, and 17 socket drops. It does not show the age of queued packets.

Filter by a known port to avoid unrelated sockets. ss uses decimal service syntax, for example ss -u -a -n -m '( sport = :9000 )'. Run it frequently enough during a load test to see peaks, while recognizing that a polling command provides samples rather than a continuous trace.

proc exposes queue size and drops

/proc/net/udp and /proc/net/udp6 list sockets visible in the current network namespace. The local_address field encodes the address and port in hexadecimal. The tx_queue:rx_queue field reports queue amounts in hexadecimal. The final field is the socket drop count on current Linux formats.

Port 9000 is hexadecimal 2328. An awk filter matching :2328 can locate the row, but production tooling should parse columns and network namespaces carefully. A container may see a different /proc view from the host. SO_REUSEPORT also creates multiple sockets on the same port, so one row is not necessarily the whole listener.

The drops count is cumulative over the socket lifetime. Capture a baseline and compute a delta. Restarting the process resets the socket identity and its counter, which can make a dashboard look repaired unless the application retains event history.

UdpRcvbufErrors is the host-wide corroboration

nstat -az UdpInDatagrams UdpInErrors UdpRcvbufErrors reads UDP MIB counters. UdpRcvbufErrors increments when a datagram cannot be queued because the UDP receive buffer is full. UdpInErrors includes a broader class of receive errors. UdpInDatagrams counts datagrams delivered to users.

These values are host or network-namespace totals rather than one-socket statistics. Correlate the UdpRcvbufErrors delta with per-socket drops and application sequence gaps. If the totals rise but the target listener remains flat, another UDP workload may be responsible.

Prometheus node exporters and custom collectors can expose these counters, but field names and namespace visibility must be verified. Alert on a positive rate during feed activity. A static nonzero total from an old incident is not a current page.

netdev backlog protects a preceding stage

net.core.netdev_max_backlog sets the maximum packet count queued on the input side when an interface receives faster than the kernel can process. It applies before the packet reaches a UDP socket. The second field of each /proc/net/softnet_stat row counts input backlog drops in hexadecimal.

net.core.netdev_budget limits packets processed in a network polling cycle, and net.core.netdev_budget_usecs sets a time budget on applicable kernels. The third softnet_stat field counts times the cycle exhausted its budget. Raising a budget can drain more network work per cycle but may delay user-space scheduling on that CPU.

A netdev_max_backlog value of 8192 is a practical test point on a dedicated receiver experiencing short bursts. It is not a default recommendation for every system. If the softnet drop field stays flat, increasing the backlog does not address socket pressure.

NIC ring depth covers hardware service gaps

ethtool -g eth0 shows supported and current receive and transmit ring sizes when the driver allows it. ethtool -G eth0 rx 4096 requests a new receive ring. The NIC must support the value, and some virtual interfaces do not expose ring controls.

A deeper ring gives the driver and NAPI poller more descriptors during a burst or CPU pause. It can also retain packets longer, increase cache footprint, and mask poor interrupt placement. Driver counters such as rx_missed_errors or rx_no_buffer_count, where available, indicate pressure before the kernel stack.

Ring changes can briefly disrupt traffic and should be tested in a maintenance context. Record the prior current value, because the maximum is not necessarily the vendor default or the best latency setting. A full ring under steady load requires more receive service capacity, not an infinite descriptor count.

Large buffers can hide stale data

At 54.3 Mbps, the measured packet-byte rate is about 6.79 MB/s. A 64 MiB socket allocation can represent many seconds of payload, with the actual duration reduced by socket-memory overhead. Processing all queued shreds after a long pause can violate a strategy's freshness requirement even if no datagram was dropped.

Packet timestamps expose residence time. SO_TIMESTAMPNS adds a software timestamp through recvmsg ancillary data. SO_TIMESTAMPING supports broader software and hardware timestamp options when the NIC, driver, and clock configuration permit them. Compare receive timestamp with the monotonic time at dequeue or decode using compatible clock domains.

Set an age threshold and a protocol-aware shedding rule. The safe action may be to invalidate a partial fec set, discard old queued work, or keep receiving while recovery occurs on another core. A large buffer needs an equally explicit stale-data policy.

Application drain rate is the durable fix

The receive loop should batch datagrams with recvmmsg, reuse buffers, avoid synchronous logs, and perform only bounded validation before handoff. Hash maps, allocations, signatures, erasure recovery, and transaction decoding can move to workers. The handoff must remain bounded and observable.

Measure sustained packets per second per ingest core above the production rate. The feed mean is 5,585 packets/sec, which is not difficult for a well-written native receiver, but runtime pauses, shared CPUs, and expensive per-packet work can create bursts of under-service. Test at higher rates and with real packet sizes.

If the application is slower than arrival indefinitely, no finite kernel value is correct. Adding memory changes when loss begins. Improving service rate or shedding work changes whether the queue returns to empty.

Tune with a controlled pause test

A useful buffer test has a numbered source, the production packet-size distribution, and an instrumented receiver. Establish zero loss at steady state. Pause only the consumer for 10, 50, 100, and 250 ms while the sender continues. For each pause, record maximum socket allocation, UdpRcvbufErrors, per-socket drops, application gaps, and maximum age after resume.

The chosen setting passes only if it absorbs the agreed pause and the receiver catches up within the age budget. A setting that avoids loss but requires two seconds to clear is not acceptable for a 50 ms freshness target. A smaller buffer may make failure explicit sooner and be safer for some strategies.

Repeat after kernel, driver, instance type, CPU-affinity, and receiver-build changes. Buffer capacity is an observed system property. Treating it as a copied constant invites regression.