Busy polling and NAPI for UDP latency

NAPI keeps packet interrupts bounded

Linux network drivers commonly use NAPI to combine interrupt notification with polling. A receive interrupt signals work. The kernel schedules a NAPI poll, temporarily masks or suppresses further receive interrupts for that queue, and processes a bounded batch of packets. If work remains, polling continues according to kernel budgets. When the ring is drained, interrupts are re-enabled.

This design avoids an interrupt for every packet during load. It also creates a scheduling path between NIC arrival and socket delivery. Queue placement, poll budget, interrupt moderation, CPU contention, and application wakeup all influence the resulting latency distribution.

NAPI is not an application setting that should be disabled for lower latency. It is the normal high-performance receive mechanism. The tuning problem is to keep the right queue and CPU responsive, process enough work per cycle, and avoid starving the receiver or other tasks. Busy polling adds a controlled application-driven poll opportunity to this path.

Interrupt mode and poll mode alternate

At low traffic, a receive queue often relies on an interrupt to begin work. The interrupt handler schedules software processing, and the NAPI poll drains packets. Under sustained traffic, the poller may find more work each cycle and remain active, reducing interrupt frequency. Under bursty traffic, transitions between idle and active introduce wakeup cost.

The precise sequence depends on driver and kernel implementation. Hardware interrupt moderation can delay notification so several packets arrive before one interrupt. Software budgets can stop processing even while descriptors remain. Receive-side scaling distributes queues across CPUs. Generic receive offload may combine stack work.

Measure the deployed path instead of assuming a diagram gives its latency. /proc/interrupts shows IRQ counts per CPU. /proc/net/softnet_stat shows processed packets, drops, and budget exhaustion per CPU. ethtool -c eth0 shows interrupt-coalescing settings where supported. Together they explain more than one average packet timestamp.

Busy polling asks user space to spend CPU

Socket busy polling lets a blocking receive path poll the associated NAPI context for a bounded number of microseconds before sleeping. If a packet arrives during that window, the application can receive it without a full interrupt and scheduler wakeup path. The trade is direct: lower idle-to-packet latency can cost substantial CPU and energy.

SO_BUSY_POLL configures a per-socket busy-poll time in microseconds on Linux. net.core.busy_read supplies a default for blocking socket reads when supported. net.core.busy_poll supplies a default used by poll and select paths. Kernel support, driver NAPI identifiers, and socket association are required for an effect.

A value of 50 means a busy-poll opportunity measured in tens of microseconds, not a guaranteed 50 microsecond latency reduction. If the receive thread is on the wrong CPU, shares a busy core, or uses a driver without support, the result can be neutral or worse.

The socket must be associated with a receive queue

Linux learns a socket's NAPI association from packets received through a compatible device and queue. Busy polling can then invoke polling for that context. A socket with no prior packet or a path without the required driver support may sleep as usual. Network namespaces, virtual devices, and tunnels can change which NAPI context is visible.

SO_INCOMING_CPU and SO_INCOMING_NAPI_ID can help an application inspect receive placement on supported kernels. SO_INCOMING_CPU can also express a preferred CPU for incoming processing in some designs. These options are diagnostic and steering tools, not a replacement for configuring RSS, IRQ affinity, or one listener per queue.

Busy polling works best when the receive thread, memory, queue interrupt, and NAPI work have intentional locality. Polling a remote queue can add cache-line movement. One global sysctl applied to every socket can spend CPU across unrelated services without improving the critical flow.

Global and per-socket controls have different scope

net.core.busy_read and net.core.busy_poll are expressed in microseconds. A system can expose them through sysctl, although kernel builds and distributions may omit or default them to zero. A value of zero disables the relevant default busy-poll behavior.

SO_BUSY_POLL is preferable for a targeted receiver because it limits the change to one socket. The process uses setsockopt before entering its receive loop and checks the return value. Global defaults can be useful on a dedicated appliance where every relevant socket has been reviewed.

Newer Linux versions provide additional controls such as SO_PREFER_BUSY_POLL and SO_BUSY_POLL_BUDGET for persistent preference and budget behavior under appropriate privileges. Their exact availability and privilege rules are kernel-version dependent. Gate them through compile-time and runtime checks, and keep a fallback to normal blocking receive.

Polling mode can consume an entire core

A thread repeatedly calling nonblocking recvmsg in a tight loop is application spin polling. SO_BUSY_POLL is kernel-assisted polling during socket operations. Both can approach full core utilization when traffic is sparse. Their behavior and accounting differ, but the capacity cost is real.

On a dedicated latency receiver, one core per ingest queue may be an acceptable price. On a shared machine, busy polling can steal cycles from decoding, increase thermal load, reduce turbo headroom, and worsen other network queues. Cloud schedulers may also throttle or move virtual CPUs in ways that erase the benefit.

Measure CPU time per packet at both ordinary and quiet rates. A setting that improves p50 by 3 microseconds while adding one saturated core and no p99.9 gain may be a poor trade. A setting that removes a 100 microsecond wakeup tail during sparse critical traffic may be worthwhile.

NAPI budgets bound each processing cycle

net.core.netdev_budget limits the number of packets processed in one networking softirq cycle across participating work. net.core.netdev_budget_usecs limits time spent in that cycle on kernels that expose it. When the budget is exhausted with work remaining, processing resumes later. The third field of /proc/net/softnet_stat records time-squeeze events.

Increasing budgets can reduce queue accumulation under bursts. It can also keep a CPU inside networking work longer, delaying application threads scheduled on that CPU. Pinning an ingest thread to the same CPU as a heavily loaded softirq can therefore create competition between producing socket data and consuming it.

Budget changes require a paired measurement of softnet time-squeeze deltas, socket drops, receiver scheduling latency, and packet age. A lower softnet counter with a slower application is not an improvement. Kernel version and workload shape affect the balance.

Interrupt coalescing sets an earlier delay

NIC interrupt moderation waits for a packet count, time threshold, or adaptive rule before notifying the CPU. ethtool -c eth0 reports fields such as rx-usecs, rx-frames, and adaptive-rx where the driver supports them. ethtool -C changes those values.

Lower rx-usecs can reduce time to the first packet in a burst but increases interrupt rate. A setting of rx-usecs 0 may produce more predictable low load latency on a dedicated queue, yet overwhelm a CPU under heavy mixed traffic. Adaptive moderation changes behavior with load and can create a distribution that is harder to reason about.

Busy polling may find packets before a moderated interrupt fires, which is one source of its latency benefit. It does not remove the need to configure coalescing. Test a small matrix of busy-poll and coalescing values because their effects interact.

RSS and IRQ affinity establish locality

Receive-side scaling hashes flows into hardware receive queues. ethtool -l eth0 reports channel counts, ethtool -x eth0 reports the indirection table on supported devices, and /proc/interrupts shows the IRQs associated with queue names. /proc/irq/IRQ/smp_affinity_list controls allowed CPUs for an IRQ.

One UDP flow normally hashes to one receive queue, so a single destination tuple may use only one hardware queue. Multiple SO_REUSEPORT sockets with appropriate steering can distribute multiple flows, but cannot manufacture flow diversity from an invariant five-tuple without driver or BPF support.

Place the receive thread near the selected queue's CPU and memory. Avoid assigning unrelated high-rate IRQs to that CPU. irqbalance may rewrite manual affinity, so either configure it with exclusions or manage IRQ placement through a persistent service. Verify after reboot and interface reset.

GRO changes the number of stack events

Generic receive offload aggregates compatible packets to reduce per-packet stack work. UDP GRO support exists on modern Linux paths but depends on driver, kernel, and socket behavior. ethtool -k eth0 shows generic-receive-offload and related feature state.

Aggregation can improve throughput and reduce NAPI pressure. It can also delay delivery until a flush condition and change what capture tools observe. A receiver that assumes each returned buffer maps to one original datagram needs to handle any segmentation metadata exposed by its API and selected offload path.

For a 5,585 packets/sec shred feed, ordinary kernel UDP is often sufficient without aggressive aggregation. Test GRO as a separate variable. Correctness, individual shred boundaries, and p99.9 age matter more than a reduction in reported packets processed by the stack.

Use busy polling only after removing obvious stalls

Busy polling cannot repair a receive loop that performs synchronous logging, allocates excessively, or blocks on a full worker queue. It cannot fix NIC drops, an incorrect RSS mapping, or a decoder with lower sustained throughput than arrival. Those failures remain and may become harder to see under higher CPU consumption.

First establish clean counters and bounded queues under normal blocking receive. Batch with recvmmsg, pin intentionally, preallocate buffers, and isolate expensive work. Then compare busy polling against that credible baseline.

This order matters because busy polling often produces a small latency gain. A 200 microsecond logging stall dwarfs it. Operator depth means removing the large unforced errors before tuning the kernel's wakeup path.

Measure receive timestamps and CPU together

Use hardware receive timestamps when the NIC, driver, and clock setup support them. Otherwise, SO_TIMESTAMPNS provides a software receive timestamp taken within the kernel path. Compare it with a monotonic application timestamp only when clock domains are compatible and understood.

Record p50, p99, p99.9, and maximum arrival-to-user latency, plus CPU utilization, context switches, migrations, IRQ rate, NAPI budget exhaustion, UdpRcvbufErrors, and application queue age. Run sparse and sustained traffic profiles. Busy polling may help sparse traffic and provide little change when NAPI is already continuously polling under load.

Use paired runs on the same host with fixed CPU frequency policy and affinity. Warm up memory and code paths. Retain the full distributions. One minimum-latency number is especially vulnerable to noise and says little about production tails.

A shred receiver needs a narrow experiment

The measured feed averages 5,585 packets/sec, which means an average inter-arrival interval near 179 microseconds. Arrival is bursty, so the mean is not a scheduling promise. A 50 microsecond busy-poll window can bridge some gaps without necessarily spinning continuously, but actual CPU duty depends on receive-loop behavior and bursts.

Start with SO_BUSY_POLL=50 on the one ingest socket, not a large global value. Compare zero, 25, 50, and 100 microseconds. Keep interrupt moderation fixed, then repeat the most promising cases with a measured coalescing change.

Kernel bypass is not the automatic next step if busy polling fails to help. At this packet rate, a tuned kernel path has substantial headroom on modern hardware. AF_XDP or DPDK may reduce variance, but they add queue ownership, memory, deployment, and observability complexity. Use them only after the required tail cannot be reached through the ordinary path.