XDP and kernel bypass for UDP receivers

XDP runs before the ordinary network stack

Express Data Path, or XDP, runs an eBPF program at an early receive hook. In native driver mode, the program executes after the driver has a packet buffer but before the kernel allocates the ordinary socket-buffer path. It can pass, drop, transmit, or redirect the packet with actions such as XDP_PASS, XDP_DROP, XDP_TX, and XDP_REDIRECT.

This placement is valuable for filtering and steering. An XDP program can discard irrelevant traffic before UDP socket lookup, count malformed lengths, or redirect selected packets into an AF_XDP socket. Packets returned as XDP_PASS continue through the normal Linux stack.

XDP itself is not synonymous with kernel bypass. A program that returns XDP_PASS still uses IP, UDP, socket queues, and normal system calls. It has added an early programmable stage. AF_XDP provides the path that can redirect frames into user-managed memory and avoid much of the conventional stack.

Native, generic, and offload modes differ

Native XDP runs in the NIC driver's receive path and usually provides the intended performance. Generic XDP runs later through the skb path when a driver lacks native support. It preserves much of the programming model but does not provide the same early execution or allocation savings. Hardware offload runs a supported program on the NIC itself.

Driver and device support determine available modes, helpers, metadata, multi-buffer behavior, and zero-copy AF_XDP. ethtool -i eth0 identifies the driver and firmware. bpftool feature probe inspects kernel BPF capabilities. ip -details link show dev eth0 reports attached XDP state.

Do not report one XDP benchmark without its mode. A generic-mode result on a virtual interface and a native zero-copy result on a physical queue describe different systems. Deployment tooling should require the expected mode instead of falling back silently.

AF_XDP connects a queue to user memory

AF_XDP is an address family optimized for high-rate packet processing. The application registers a UMEM region containing fixed-size frames, creates fill and completion rings for buffer ownership, and uses receive and transmit rings for descriptors. An XDP program redirects matching packets through an XSKMAP to an AF_XDP socket.

In zero-copy mode, a compatible driver and NIC can DMA received data into UMEM frames without copying through ordinary socket buffers. Copy mode remains available on more devices and can still avoid portions of the protocol stack. The application polls descriptors, parses Ethernet and IP headers, validates UDP, and returns buffers to the fill ring.

The gain comes from less general-purpose work and direct queue ownership. The cost is that user code now owns buffer lifecycle, frame sizing, ring replenishment, wakeups, parsing, and more failure states. A stalled fill ring means the NIC path has nowhere to place packets.

DPDK bypasses more of Linux networking

The Data Plane Development Kit provides user-space poll-mode drivers, hugepage-backed memory, lockless rings, and packet-processing libraries. A DPDK process commonly binds a NIC or virtual function to vfio-pci, polls hardware queues on dedicated cores, and handles frames without the ordinary Linux network stack.

This can deliver high packet rates and predictable polling latency. It can also remove the interface from normal host networking. SSH, monitoring, firewalling, routing, and other services need a separate management interface or carefully partitioned queues and functions. Hugepage and IOMMU configuration become deployment dependencies.

DPDK is not always worth the complexity. A measured 5,585 packets/sec feed is modest for a tuned kernel UDP receiver. Decoder and strategy work may dominate. Taking full queue ownership to save a small receive-path cost can make upgrades, observability, and incident response materially harder without changing the business latency.

Packet rate decides when the stack matters

Kernel overhead is mostly paid per packet, so packets per second matters alongside bandwidth. The feed averages 54.3 Mbps and 5,585 packets/sec with a 1,216 byte mean packet. Modern Linux can process far higher packet rates on suitable hardware when IRQ affinity, batching, buffers, and application work are disciplined.

Microbenchmarks at 64 byte packets and tens of millions of packets per second demonstrate a bypass ceiling, not this workload. Reproduce production packet sizes, burst structure, CPU topology, and downstream work. The relevant metric is receive-to-decision tail latency with correct recovery, not the maximum rate of a loop that drops every payload after counting it.

Kernel bypass becomes credible after ordinary-path evidence shows a bottleneck. Examples include persistent softnet budget exhaustion, socket-path CPU cost that consumes an ingest core at required burst rate, or a wakeup tail that busy polling cannot meet. Preference is not evidence.

XDP is valuable before full bypass

An XDP filter can reduce hostile or irrelevant traffic before it consumes socket and stack work. A shred receiver can accept the intended Ethernet type, IP protocol, destination address and port, expected source IP 64.130.40.90, and allowed packet-length range, then pass valid candidates to UDP.

This early filter complements, but does not replace, nftables and application validation. Source addresses can be spoofed on some paths. XDP code must handle VLANs, IPv4 options, IPv6 extension headers, fragments, and short frames safely or intentionally pass them to a slower path. A narrow parser can create an outage during a legitimate network change.

XDP maps can count actions per CPU without a log per packet. Aggregate map values in user space and export bounded metrics. Map lookup and update cost should remain small, and operators need a way to detach the program if its control process fails.

Count every terminal action, including PASS, DROP, TX, REDIRECT, ABORTED, and any parser-specific reason. XDP_ABORTED represents an exception-like program outcome and should be near zero. Tracepoints can diagnose it during a controlled incident, but continuous tracing at feed rate is excessive. Per-CPU maps avoid a shared atomic counter on the packet path.

Queue steering determines the AF_XDP layout

An AF_XDP socket binds to an interface and queue ID. The application normally creates one socket and polling thread per selected hardware receive queue. RSS or hardware flow steering must send target packets to those queues, and the XDP program must redirect to the matching XSKMAP entry.

One stable UDP five-tuple often hashes to one queue. Creating eight AF_XDP sockets does not distribute that flow across eight hardware queues. ethtool -x eth0 and ethtool -n eth0 rx-flow-hash udp4 reveal supported RSS configuration. ethtool -N can install hardware receive-flow rules on compatible drivers, but rules select a queue rather than round-robin one flow.

The receive queue, polling thread, UMEM, and decoder handoff should share NUMA locality. A zero-copy descriptor whose processing bounces to another socket can lose much of the benefit through cache and memory traffic.

Buffer ownership replaces socket-buffer tuning

An ordinary UDP receiver sizes SO_RCVBUF and observes UdpRcvbufErrors. An AF_XDP receiver sizes UMEM frames, fill-ring entries, receive-ring entries, and application queues. It must keep the fill ring supplied. Exhaustion and invalid descriptors are application-visible conditions rather than UDP socket drops.

Frame size must hold the complete received packet plus required headroom. A 2,048 byte UMEM frame can hold an ordinary Ethernet frame carrying a maximum 1,228 byte UDP payload with room for common headers. Multi-buffer packet support varies, so relying on chained frames requires exact kernel and driver validation.

Use hugepages where the chosen design and library benefit, pin memory, and budget it per queue. Preallocate all data-plane objects. A dynamic allocation or page fault in the poll loop undermines the predictability sought from bypass.

Polling and need-wakeup trade CPU for latency

AF_XDP supports polling modes and an XDP_USE_NEED_WAKEUP binding flag. In need-wakeup mode, ring flags indicate when the application must invoke poll or sendto to wake kernel or driver work. Correct handling reduces unnecessary system calls while avoiding a stalled ring.

A pure busy loop checks receive descriptors continuously and can consume one core per queue. Interrupt-driven poll can save CPU but restores wakeup latency. Adaptive loops spin for a bounded interval, then block. The same trade appears in SO_BUSY_POLL, but AF_XDP exposes lower-level rings and ownership.

Record empty poll iterations, packets per batch, wakeup calls, ring occupancy, and CPU cycles. An apparent p50 improvement purchased with eight spinning queues for one active flow is an architecture error, not a tuning success.

The application must rebuild protocol checks

AF_XDP and DPDK receive Ethernet frames, not validated UDP payloads. The application must parse VLAN tags, address family, IP header length, fragmentation state, total length, UDP length, destination, and checksum policy. IPv6 extension headers require bounded traversal. Truncated or malformed frames must be rejected before payload access.

The ordinary Linux stack has years of hardening around these cases. Reimplementing only the happy path expands the attack and outage surface. Source filtering at the edge does not eliminate malformed input, especially on shared or cloud networks.

Fuzz the parser with short frames, overlapping length claims, IPv4 options, fragments, VLAN stacks, bad checksums, and unexpected packet sizes. Compare accepted payloads against the normal UDP path in a shadow test. A bypass receiver is network protocol software and needs that engineering standard.

Observability changes below the socket layer

UdpInDatagrams and /proc/net/udp no longer describe packets redirected away from UDP into AF_XDP. Socket drops can stay flat while AF_XDP fill-ring exhaustion loses traffic. Existing host dashboards may therefore turn green as the new receiver fails.

Expose XDP action counters, XSK redirect success, invalid redirect results, fill-ring starvation, RX ring occupancy, user-ring rejects, parse failures, sequence gaps, and packet age. bpftool prog show and bpftool map show confirm attachment and map existence. bpftool net lists network BPF attachments on supported versions.

Driver counters remain relevant. ethtool -S eth0 can reveal queue misses and drops before XDP. Hardware timestamp access and semantics may differ on AF_XDP. Validate the actual timestamps instead of reusing a normal-socket latency dashboard unchanged.

Deployment and rollback need their own design

Attaching an XDP program can affect every packet on an interface. A verifier rejects unsafe bytecode, but a logically wrong verified program can drop management traffic. Use an interface dedicated to the feed where possible. Keep management connectivity separate and provide an out-of-band rollback.

Program replacement can use pinned maps and link-based attachment patterns to reduce gaps, depending on loader and kernel. The control plane should verify program ID, tag, expected mode, map schema, and queue bindings after every restart. A loaded process is not proof that XSKMAP entries are populated.

For DPDK, rollback may require stopping the process, rebinding the device to its kernel driver, restoring addresses and routes, and restarting network services. Rehearse that sequence before production. It is a material operational tax.

Compare three paths with one acceptance test

Build an ordinary recvmmsg receiver first. Add socket busy polling as a second variant. Add AF_XDP only as a third. Feed all variants the same captured or generated numbered packets at production and burst rates. Keep decoder work equivalent and validate byte-for-byte accepted payloads.

Measure p50 through maximum latency, loss location, cycles per packet, cores consumed, memory, deployment steps, restart behavior, and monitoring coverage. Include sparse traffic because busy polling and bypass have different idle costs. Include a malformed corpus because a fast incorrect parser has no value.

Choose the least complex path that meets the required tail with headroom. Kernel bypass is an engineering trade, not an automatic maturity stage. For this feed rate, a conventional socket path often wins on total system reliability.